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1. WHY: The Monitoring and Evaiuation Framework - An overview 


1.1. Introducing the Monitoring and Evaluation (M&E) Framework 

This document serves as a ‘framework’ for improved monitoring and evaluation (M&E) within 
the City of Johannesburg Group - i.e. within all Municipal Entities (MEs) and internal 
departments (collectively referred to, hereafter, as ‘the City’ or ‘CoJ’). It provides the 
foundation for a common understanding of key M&E principles and elements amongst all 
role-players in the City. 

While M&E activities are as applicable to the domain of individual performance management 
as they are to group-wide, cluster or departmental performance management, the primary 
focus here is on M&E in the context of the latter areas. Included in this framework is an 
overview of the City’s current context and the rationale for establishing a more rigorous M&E 
approach, clarity on the key concepts included in the M&E Framework, and input on how 
other spheres of government and delivery agents external to government address this issue. 
Also addressed are learnings emerging on the requirements for a successful M&E approach. 
Following this input on the ‘why and ‘what of M&E, emphasis is placed on the ‘nuts and 
bolts’ of the framework itself (e.g. activities and timeframes), and roles and responsibilities in 
relation to the framework. Comment is also included on the next steps deemed necessary 
for the system to be implemented effectively - e.g. alignment with other systems and 
processes in place within the City and within other spheres of government. 

These latter elements therefore address the ‘when’, ‘hoW and ‘who’ of M&E, laying the 
groundwork for a practical M&E Handbook, to be finalised following the sign-off of this M&E 
Framework. The Handbook will provide step-by-step guidelines for implementation of M&E 
practice - and will include all templates, checklists and toolkits necessary for the full rollout 
of a revised M&E approach. 


1.2. Developing the M&E Framework 

The M&E Framework included here is based on the review of a range of relevant documents 
- including those relating to existing planning, performance management, monitoring, 
evaluation, reporting and governance-related practices and policies in place within the City. 
It builds on related frameworks developed by the City to date, including the current Group 
Performance Management Framework. The shift in approach takes into account the 
increased levels of engagement expected by the City’s stakeholders (including citizens and 
residents) - and the requirements for meaningful M&E within the context of the City’s rapidly 
changing form and environment. 

In developing the framework, international and national M&E benchmarks were considered, 
alongside legislative requirements relating to the field of M&E. The ‘National Evaluation 
Policy Framework’ (NEPF), as finalised by the DPME in November 2011 served as a 
valuable input, alongside a range of preceding guidelines such as the Presidency’s 2010 
‘Guide to the Outcomes Approach’, and more specifically. Outcome 9. 


The City of Johannesburg’s Monitoring and Evaluation Framework 

1 I Page 


14 May 2012 



In addition to focusing on technical documents relating to M&E, national and City-specific 
strategies such as the National Planning Commission’s (NPC’s) recently published National 
Development Plan (NDP) and the City’s ‘Joburg 2040’ Growth and Development Strategy 
(GDS) were considered. Efforts focused on ensuring alignment with the key principles 
included in these strategies. Analysis of the concepts and approaches reflected across all 
documents was synthesised into a framework for the City, for further engagement and 
refinement, with this framework also submitted to the DPME for comment and input, prior to 
finalisation. 


1 .3. Structure of the M&E Framework 

The M&E Framework included here is structured as follows: 

• Chapter 1 presents an introduction to the M&E Framework - with comment on the 
context and rationale for development, applicability and the steps followed in 
establishing the M&E Framework itself. 

• Chapter 2 provides detail in terms of the key concepts relating to M&E. 

• Chapter 3 outlines the approaches adopted by other role-players in respect of M&E 
- with focus placed on approaches within the global arena, alongside the direction 
followed at a national and provincial government level, and those approaches 
followed to date by the City itself. Also included are learnings relating to the 
requirements for an effective M&E Framework, drawn from experiences of M&E 
application within other contexts. 

• Chapter 4 provides input on the City’s context in respect of M&E, highlighting the 
principles and values underpinning the City’s M&E Framework. Input is also provided 
on the GDS (in particular, the vision, outcomes and principles included within it), and 
the mechanisms through which this is supported via M&E. The GDS’ role as the 
City’s primary ‘beacon’ for future delivery - and therefore its place as the foundation 
for M&E activities - is addressed. Input is provided on the application of M&E 
activities to short, medium and long-term planning mechanisms - in the context of 
the outcomes approach. 

• Chapter 5 outlines the key activities relating to rollout of the M&E Framework itself, 
drawing on insights included within the preceding chapters. The overview of each of 
the key steps involved in the M&E process is supported with a more detailed 
explanation of delivery. Practical examples are included, together with a set of 
templates - with these to be bolstered in the more practical application-focused M&E 
Handbook, to follow. 

• Chapter 6 identifies the governance and oversight issues that emerge in the context 
of the M&E Framework - with input included on details relating to roles and 
responsibilities, reporting arrangements and other related factors. 

• Chapter 7 provides an overview of the next steps viewed as necessary for sound 
implementation of the M&E Framework, if the overarching benefits we hope to derive 
through implementation are to be realised. 

This is augmented with a set of annexures - structured as follows: 

• Annexure 1 provides a list of acronyms and abbreviations. 
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• Annexure 2 focuses on roles and responsibilities in the context of the City’s M&E 
approach. 

• Annexures 3 to 8 include M&E Framework templates - and a practical example of 
template use. 

1 .4. Rationale for an M&E Framework 

The City has carried out M&E activities as part of its performance management process for 
many years. However, new priorities, challenges and changes within the City, across other 
spheres of government, and within broader society have heightened awareness of the 
importance of meaningful M&E processes. The City’s policies, procedures and systems 
associated with the inter-related areas of performance management, monitoring, evaluation 
and reporting have changed significantly since the point of initial implementation. The wider 
set of policies, procedures and systems associated with performance management - the 
domain within which M&E activities fall - have shifted over time due to the following needs, 
amongst others: 

• The imperative to deliver to the City’s citizens and to meet commitments made to its 
wider spectrum of stakeholders - adapting methods for delivery where necessary; 

• Managerial requirements identified as necessary for successful delivery of duties - 
e.g. accurate, data-rich information, and ongoing tracking of delivery, to support 
continuous and improved provision of goods and services; 

• The need to account and engage in two-way discussion with various stakeholders; 

• Good practice requirements and legislative and regulatory amendments, 
necessitating improvements - e.g. to governance arrangements and performance 
audit approaches; and 

• Changes resulting from revisions to the City’s institutional model, with each 
amendment leading to shifts in roles and responsibilities, reporting lines and 
supporting systems such as those associated with M&E. 

In the City’s efforts to embed performance management practices and with this, practices 
related to M&E, a number of challenges have emerged - with each holding the potential to 
undermine the original intentions of these related systems. While many of the challenges are 
associated with the broader Performance Management Framework itself, a few of the issues 
specific to M&E include the following: 

• The tendency for many to focus primarily on the link between individual performance 
and reward when establishing plans of action for the year ahead, with the resulting 
risk that the content of plans is potentially driven by reward as the end-focus, rather 
than the need to ensure alignment with longer-term outputs and outcomes necessary 
for delivery on the City-wide strategy. With this comes a challenge for M&E activities: 
a focus on indicators and targets that do not align with the organisation’s long-term 
direction or objectives. 

• Linked to the above, a particular challenge relates to the question of whether all 
individual scorecards, together, align with the outcomes defined in business plans, 
the City’s medium-term plan, the IDP, and long-term plans such as the GDS. Losing 
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track of this long-term focus means, ultimately, a misalignment between daily 
activities and the objectives committed to through public engagement. 

• The need for a greater variety of information sources to inform evaluation - e.g. input 
from ward councillors, or input via a 360-degree appraisal. 

While the above requires a more holistic focus on performance management and 
governance, the City has chosen to hone in on M&E, given the potential short, medium and 
long-term delivery-related consequences that may result from successful improvement to 
this element. As noted, amendments reflected within this M&E Framework will be woven 
through a set of supporting guidelines and systems, to ensure sound application. 

Nationally, the benefits to be realised from effective and efficient M&E activities - particularly 
in supporting governance and delivery-related objectives - have become more apparent. 
This has led to the establishment of a fully-fledged Department of Performance, Monitoring 
and Evaluation (DPME) in the Presidency, and a range of policies and guidelines through 
which to drive M&E and ultimately, the delivery of desired outcomes across all three spheres 
of government. At a CoJ level, changes to the organisation’s institutional model and its long- 
term Joburg 2040 GDS have necessitated a review of the way in which the planning, 
decision-making, monitoring and evaluation of delivery takes place. The City has 
acknowledged the need for improvements to its internal processes, to ensure effective 
rollout of the GDS through greater alignment between the five-year and one-year IDPs and 
the GDS’ longer-term goals. As a result, the core priorities underpinning the GDS (i.e. 
establishment of sustainable services and settlements; resource security and environmental 
sustainability; an inclusive economy; eradication of poverty and enhanced social inclusion) 
have been used as the foundation for a refined governance arrangement in the City. This 
has taken the form of a Cluster Mayoral Sub-committee system. Four Mayoral Sub- 
committees were established in November 2011, addressing the clusters of: ‘sustainable 
services’; ‘economic growth’; ‘human and social development’ and ‘good governance’. 

Through developing this framework, the City aims to establish: 

• A City-wide understanding of M&E; 

• A common, standardized language and approach for the application of M&E 
principles across the City as a whole; 

• Enhanced M&E practices - in terms of M&E methodology and tools, and the quality, 
frequency and application of findings; 

• Clarity in relation to the roles and responsibilities of all those who are directly or 
indirectly involved in M&E activities; 

• The means through which to institutionalise M&E - and ensure application of 
learnings arising from analysis, for improved delivery; 

• A mechanism for greater integration of M&E practices within the City’s public 
participation, planning, budgeting, delivery, policy development, oversight, reporting 
and governance-related processes; and 

• Greater transparency and accountability, through the generation of sound information 
- to be used in reporting, communication and the improvement of delivery. 
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1.5. Principles underpinning M&E 


The DPME’s policy framework outlines a set of principles - drawn from the Policy 
Framework for Government-wide Monitoring and Evaluation System (GWMES) - that are 
viewed as important for the practice of evaluation (although these may be seen as applicable 
to both M&E). These are depicted in the box below. 

Box 1. 1: Principles guiding M&E (Adapted from GWMES, cited in DPME, 2011, p.3) 

M&E: 

1 . “. . .should be development-oriented and should address key development priorities 

of government and of citizens. 

2. ... should be undertaken ethically and with integrity. 

3. ... should be utilisation-oriented. 

4. ... methods should be sound. 

5. ... should advance government’s transparency and accountability. 

6. ... must be undertaken in a way which is inclusive and participatory. 

7. ... must promote learning.” 


The City has chosen to adopt these principles, ensuring delivery through the design and 
application of the M&E Framework. 

Many of the elements included within these principles relate closely to the concept of ‘good 
governance’ - where this is, according to King III, “...essentially about effective, responsible 
leadership”, with responsible leadership defined further as being “characterised by the 
ethical values of responsibility, accountability, fairness and transparency” (loD, 2009, 
p.20). In the interests of ensuring a common understanding amongst all within the City, the 
terms ‘responsible’ and ‘accountable’ are explained further below: 

• Responsible: The person tasked with carrying out or doing an activity or addressing 
a deliverable - i.e. the ‘doer’; and 

• Accountable: The person who signs off or approves work carried out by the 
‘responsible’ party - and who is held to account in the event that the task or activity is 
not completed optimally. 

Across all levels of an organisation such as the City, certain role-players are responsible for 
carrying out delivery, while others are accountable for those they are tasked with managing, 
and the activities completed or results realised. An M&E system within the public sector 
seeks to hold individuals and organisations to their commitments, with the ultimate objective 
of ensuring development priorities are met. 

In addition, in line with the principles detailed above and the recognised importance of 
ensuring delivery on areas of responsibility or accountability, consideration must be given to 
the needs and interests of all stakeholders (i.e. thereby upholding fairness), while adhering 
to the principle of open and honest communication (i.e. ensuring transparency). 


The City of Johannesburg’s Monitoring and Evaluation Framework 

5 I Page 


14 May 2012 



A well-functioning M&E mechanism has to align strongly to the principles defined above, if it 
is to enable the City to: 


• Effectively gather, analyse, track and use accurate delivery-related information in 
decision-making, thereby support achievement of Council-approved strategies such 
as the Joburg 2040 GDS, and the aligned operational plans (in the form of the City’s 
IDP, and its related planning tools); 

• Guarantee the quality of information informing decisions and actions taken by 
Councillors, managers and employees, or the quality of commitments made; 

• Hold people responsible, or accountable, for delivery against commitments made; 

• Ensure optimal delivery against plans, with the real risk emerging of stakeholder 
expectations not being met; or 

• Ultimately, uphold the requirements for good governance. 

Activities relating to M&E therefore serve as a key support for sound governance - and as 
an enabler of effective, responsible leadership. M&E activities also serve to support the 
objective of ‘oversight’ - a “key function of good governance rather than implementation”. 
Oversight “ensures that activities are implemented as planned by providing strategic 
direction ..., ensuring policies and procedures are met, instituting financial controls (including 
independent audits), and following through on key recommendations”. 


1 .6. M&E in the context of performance management 

The activities associated with M&E relate strongly to the domain of performance 
management - forming a key part of the cycle of reflection, and thereby enabling 
performance improvement. The DPME (2011, p.ii) notes that: “If we are to improve our 
performance we have to reflect on what we are doing, what we are achieving against what 
we set out to achieve, and why unexpected results are occurring. We cannot advance 
without making mistakes along the way, but we must evaluate and learn from our successes 
and our mistakes.” Improvements in performance depend on the insight that comes from 
ongoing reflection and regular assessments. This is the purpose of M&E activities. 

M&E activities form a critical part of the City’s performance management system. Beyond 
addressing compliance requirements, the City notes that an effective performance 
management system allows a municipality to “plan for performance management and 
monitor, review and improve the implementation of the municipality’s IDP”. The system 
should enable “increased accountability” and “learning and improvement”, while also serving 
as an early warning system, notifying leaders and managers of potential risks that may 
threaten achievement of the IDP (CoJ, 2009b, p.6). 

Performance management itself is viewed as “...a strategic management technique that 
links various organisational elements of performance to ...strategic outcomes of the city and 
assists ... monitor[ing] and evaluat[ion of]... performance in relation to... strategies and 
plans...” (CoJ, 2009a, p.9). Activities associated with M&E are therefore essential for the 
effective management of organisational performance, and ultimately, the achievement of the 
City’s short, medium and long-term goals, as reflected through its various planning tools. 
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Monitoring and evaluation forms part of the Performance Management Framework, which 
encompasses setting performance indicators, measuring them over time, evaluating them 
periodically and finally, making course corrections as needed (see below): 

Figure 1.1: Performance Management Framework 



This is integrated with the ‘results based’ or ‘theory of change’ model, as detailed further in 
the sections that follow. This approach allows us to ascertain the role of various inputs in 
delivering on outputs, outcomes and finally impacts. 

The key to the success in M&E lies in the establishment of deep sector and programme 
understanding, knowledge and experience of setting up successful monitoring systems, and 
the development, amongst all role-players, of a keen eye to enable identification of lessons 
and areas of improvements. The City has in the past developed several monitoring tools that 
have facilitated the identification of bottlenecks in current implementation - and thereby 
framed feasible course correction recommendations for the future. Such tools include mid- 
term assessments, satisfaction surveys, and end of term assessment reports. 

The M&E Framework builds on work undertaken to date, with focus placed on establishing 
more rigorous and regular M&E mechanisms, thereby enabling the City to drive its delivery 
priorities with greater certainty of achievement. 
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2. WHAT: Key concepts included in the M&E Framework 


The chapter that follows explores key concepts of relevance to the City’s M&E Framework, 
with focus first placed on the terms ‘monitoring’ and ‘evaluation’ themselves, and their 
relationship to the domain of organisational performance management. Also included here is 
clarity on the elements included within the Presidency’s ‘outcomes approach’, as these 
inform the City’s approach to M&E. As such, this chapter addresses the ‘what of M&E in the 
City. 


2.1 . What is ‘monitoring’? 

‘Monitoring’ refers to an ongoing process focused on the assessment of projects, 
programmes and those day-to-day activities and deliverables required for their achievement, 
with performance tracked through data collection and reviews. Monitoring allows for reai- 
time analysis of delivery against plans, providing a “continuous flow of information’’, 
and thereby enabling positive decision-making (IFAD, 1999, p.155, emphasis added). 

The type, format and frequency of data collection and analysis to be undertaken for the 
purposes of monitoring is defined during the planning phase. Analysis carried out as part of 
the monitoring process supports early identification of problems and changed circumstances 
- and corrective decision-making in respect of resources, activities, timeframes and other 
related factors. 

2.2. What is ‘evaluation’? 

In contrast, evaluation’ is time-bound and periodic in nature, taking place at particular 
intervals before (serving as a formative evaluation), during (thereby aiding improvements) or 
at the end of a project or programme (serving as a summative evaluation). The NEPF 
defines evaluation as “[t]he systematic collection and objective analysis of evidence on 
public policies, programmes, projects, functions and organizations to assess issues such 
as relevance, performance (effectiveness and efficiency), value for money, impact and 
sustainability, and to recommend ways forward’ (cited in DPME, 2011, p.vii, emphasis 
added). 

Where monitoring focuses on the ongoing tracking of projects, programmes, activities or 
deliverables, evaluation may focus on assessing different types of issues - e.g.: efficiency, 
effectiveness, relevance, impact or sustainability (The Presidency, 2007). These different 
forms of evaluation are detailed below: 

• “Efficiency tells you that the input into the work is appropriate in terms of the output. 
This could be input in terms of money, time, staff, equipment and so on. When you 
run a project and are concerned about its replicability or about going to scale ..., then 
it is very important to get the efficiency element right.” (Shapiro, 2002, p.3) 

• “Effectiveness is a measure of the extent to which a development programme or 
project achieves the specific objectives it set. If, for example, we set out to improve 


The City of Johannesburg’s Monitoring and Evaluation Framework 

8 I Page 


14 May 2012 



the qualifications of all the high school teachers in a particular area, did we 
succeed?” (Shapiro, 2002, p.3) 

• Relevance provides an indication of the extent to which the objectives are relevant to 
the needs of recipients, policy specifications, and local, regional and national 
priorities (Ministry of Foreign Affairs of Denmark, 2006). 

• “Impact tells you whether or not what you did made a difference to the problem 
situation you were trying to address. In other words, was your strategy useful? Did 
ensuring that teachers were better qualified improve the pass rate in the final year of 
school? Before you decide to get bigger, or to replicate the project elsewhere, you 
need to be sure that what you are doing makes sense in terms of the impact you 
want to achieve.” (Shapiro, 2002, p.3) 

• Sustainability serves as a measure of the extent to which the benefits arising from 
an intervention are “likely to continue after ...support has been completed. While the 
four preceding criteria concern specific ...interventions, the assessment of 
sustainability addresses the effects of the development process itself over the long 
term. For example, in a road construction project, sustainability can be measured in 
terms of whether the road is likely to be maintained, the extent to which it will be used 
and provide benefits in the future, etc. ...Far too many development initiatives tend to 
fail once the implementation phase is over... Sustainability is ... increasingly central 
... putting greater emphasis on long term perspectives and on lasting improvements.” 
(Ministry of Foreign Affairs of Denmark, 2006, pp.54-55). 

Determining what issues will be evaluated in respect of a project, programme or activity will 
depend on the life cycle of the project or programme concerned, and the rationale for the 
evaluation. For the results of an evaluation exercise to be used optimally, it is critical to 
understand the purpose of the evaluation before embarking on the process. Reviewed 
together, evaluations conducted with due consideration of aspects such as those outlined 
above should provide a clear view to the decision-maker of the appropriate route forward - 
with the necessary condition being that information is both useful and credible. 


2.3. Comparing ‘monitoring’, ‘evaiuation’ and ‘audit’ 

Activities relating to monitoring and evaluation are complementary in nature - together 
aiding in keeping plans on track, and allowing for the identification of risks or challenges, 
facilitation of improvements, and thereby, enabling ongoing active learning. When combined, 
M&E activities therefore assist in clearing the path for effective delivery of results (Lahey, 
2009). 

When differentiating monitoring from evaluation, the DPME notes that “...monitoring asks 
whether the things we planned are being done right, while evaluation is asking are we doing 
the right things, are we effective, efficient, and providing value for money, and how can do it 
better” (2011, p.3). Another function closely related to monitoring and evaluation is that of 
‘auditing’. The table below provides a view of some of the key differences between these 
three complementary activities. 
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Table 2.1: Differences between monitoring, evaluation and audit (Adapted from UNDP, 
n.d.(a), p.12and GPG, 2012, p.10) 



Monitoring 

Evaluation 

Audit 

Objective 

(Why?) 

To establish baseline 
information. 

To track changes from 
baseline conditions to 
desired outcomes. 

To identify areas requiring 
corrective action. 

To validate what results were 
achieved, and how and why 
they were or were not 
achieved. 

To refine the ‘theory of 
change’, revisit original 
assumptions and objectives 
-thereby improving learning 
and future approaches. 

To confirm that what 
is reported as 
‘delivered’ has been 
delivered. 

To provide assurance 
and demonstrate 
transparency and 
accountability to 
stakeholders. 

Focus 

(What?) 

Focuses on the outputs of 
projects, programmes, 
partnerships and 
activities, and their 
contribution to outcomes. 
Checks progress against 
plans - and areas for 
action and improvement. 

Compares planned with 
intended outcome 
achievement. 

Focuses on how and why 
outputs and strategies 
contributed to achievement 
of outcomes and impacts. 
Addresses questions of 
relevance, effectiveness, 
sustainability and change. 

Focuses on shortfalls 
in delivery and 
learnings. 

Providing 

recommendations for 
improvement of 
current and future 
projects. 

Responsibility 

(Who?) 

Internal management and 
programme/ project 
manager responsibility - 
at all levels: 

• City-wide 

• Clusters 

• Entities/ Departments 

• Mayoral Committee 

• Performance 
Management, 

Monitoring, Evaluation 
and Reporting Unit 

External evaluators and 
partners 

May also be conducted 
internally - e.g.: 

• Executive Management 
Team (EMT) 

• Mayoral Committee 

• Council 

• Performance 

Management, Monitoring, 
Evaluation and Reporting 
Unit 

Independent auditors 
- with external inputs 

Johannesburg Risk 
Assurance Services 
(JRAS) 

Timing 

(When?) 

Continuous and 
systematic 

Time-bound, periodic, in- 
depth 

Before (formative), during 
(aiding improvements) or 
after a project or programme 
(summative) 

Ex-ante (systems 
reviews), ongoing, 
and on completion 
(e.g. annual) 

Focus - in 

relation to the 

outcome 

approach 

hierarchy? 

Inputs, activities and 
outputs 

Impacts, outcomes, purpose, 
overall objectives 

Outputs vs. inputs 
(effectiveness and 
efficiency); impact; results vs. 
costs; relevance to priorities 

Inputs, activities and 
outputs 

Data sources 

Progress reports; 
Management information 
systems; performance 
management data 

Evaluation reports; 
monitoring data; primary and 
secondary data sources - 
including case studies, 
surveys and statistical data 

Progress reports, 
management 
information systems 
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3. Current M&E practices within South Africa - and beyond 


The field of M&E is not new. Various approaches to M&E have been implemented globally, 
in the interests of supporting state delivery and more specifically, ensuring sound delivery in 
the context of projects and programmes. Much work has been undertaken in the area of 
donor funded and aid-related projects and programmes, with M&E findings often serving as 
the qualifying criteria for further funding. 

The chapter that follows provides input on practices employed in relation to M&E, with focus 
first placed on international practice in the context of the outcomes approach, and learnings 
emerging from years of implementation. This is followed by input on national and local 
government efforts. 


3.1 . M&E practices outside South Africa 

Many countries have made use of practices focused on M&E. Valuable insights are 
particularly available in respect of the Canadian experience, with M&E practices instituted at 
a national level since 1969, resulting in over thirty years of experience. The system in use 
has been through many permutations. In reflecting on the rationale for introducing an M&E 
system, the following is noted: 

“The broad goal in investing in an M&E system has been to generate and use 
‘results’ information that supports the government's management agenda from the 
perspective of both ‘learning’ and ‘accountability’ in the design and delivery of 
government policies, programs and services and the use of public funds. In this way, 
performance reporting generally aims to tell a ‘performance story’ rather than simply 
reporting on a limited set of indicators in the context of an accountability ‘scorecard’. 
Further, it is recognized that evaluation serves to inform decision-making, rather than 
expecting that decisions will rest solely on the results of an evaluation study.” (Lahey, 
2009, p.9) 

The above recognises that often, understanding the ‘how’ and ‘why’ of performance is as 
important as determining what has been achieved, given that the former elements enable 
future replication of results. Building M&E activities into future decision-making allows for 
optimal benefits to be gained through any investments made. 

While many government role-players have followed the approach employed by the Canadian 
government, the domain in which M&E practices are most vigorously employed is that of aid 
organisations. Bodies such as the Australian Agency for International Development 
(AusAID), the Global Fund, the Danish International Development Agency (DANIDA) and the 
United Nations (UN) have established clear guideline documents to support and ensure 
consistent M&E. All of these incorporate a focus on similar principles and different levels of 
analysis, even though the approaches may be labelled differently - e.g. the UN’s focus on 
“results based management” (n.d., p.9) aligns with AusAid’s focus on a “results hierarchy” 
(2005, p.36). 
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The sections that follow address learnings arising from the above applications - both in 
terms of those elements deemed to be necessary for inclusion in an M&E system, and those 
steps viewed as essential for the smooth implementation of such a system. 


3.2. Learnings from international benchmarks: Requirements for sound M&E 

Over the decades, practitioners have reflected on those M&E systems implemented across 
other national contexts, and in projects, programmes, community and aid work 
environments. Certain non-negotiable elements have been identified as necessary for the 
establishment of an effective and efficient any M&E system. Critical steps have also been 
detailed in respect of implementation. Both these areas of learning are addressed below, 
providing a useful platform for the establishment of a CoJ M&E Framework - and aligned 
systems and tools. 


3.3. M&E approaches in context: An area for learning and improvement 

In reviewing M&E approaches implemented within the global environment, and within the 
national, provincial and local context, it is useful to reflect on the experience of M&E itself - 
and more specifically, the practice of the outcomes approach to M&E. While M&E 
approaches are used across all of these domains, there is an ongoing focus on improving 
techniques and methods, to ensure achievement of commitments. 

In the South African context, the outcomes approach to M&E is still fairly new - with formal 
implementation only initiated at the level of national government in 2010. The City’s efforts in 
implementing this approach are in line with intentions across the other spheres of 
government. While there are many entities from whom we can learn - particularly in the aid 
and donor-funding domain, establishing a suitable City-specific M&E approach will take time, 
effort and ongoing reflection, before the full benefits are realised. 


3.3.1. Elements deemed critical in establishing an effective, credible M&E system 

In reflecting on the Canadian experience, Lahey (2009) identifies a number of elements and 
characteristics viewed as critical for inclusion in an M&E framework or system, to drive 
credibility and effectiveness. These include: 

• Emphasis placed on both ‘M’ and ‘E’ - through, for example, the establishment of 
a balanced focus on both elements within the M&E Framework, with attention given 
to ensuring role clarity, capacity building, tools, and the provision of support in 
respect of both. In terms of roles and responsibilities, it is noted that programme, 
project and activity managers are often responsible for monitoring, while internal or 
external evaluation specialists frequently focus on evaluation. 

• Formalisation of M&E requirements - through, for example, establishing a 
centrally driven policy and the development of clear procedures for implementation. 

• A focus on capacity building - through, for example, the establishment of M&E 
guidelines, and the creation of an M&E ‘Centre of Excellence’ in respect of M&E 
approaches and methodology - to support ongoing and proactive capacity building. 
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to oversee adherence of M&E activities against specified standards, and to assess 
quality and coverage of evaluations. 

• Building clarity in terms of who conducts evaluations - through, for example, the 
identification of an internal M&E role-player or unit/s, to provide internal ‘evaluation’ 
services, and use being made of additional support via external evaluators. 

• The establishment of quality control mechanisms and activities - such as, for 
example: the periodic involvement of the Auditor General (AG) in evaluations of the 
M&E system’s efficacy; evaluation of departmental heads on how effectively M&E 
information has been used; Scheduling and conducting reviews of the system. 

• Ensuring evaluator independence - through, for example: stressing the neutrality 
of all evaluators and the M&E function, within all policy and guidelines documents; 
ensuring evaluators have unencumbered access to management and leadership; the 
establishment of an Evaluation Committee constituted of senior role-players, and 
focused on evaluation planning, oversight and follow-up. 

• Promoting transparency - through, for example, driving this principle in the policy; 
demonstrating the commitment to this principle through publishing evaluation findings 
on the organisation’s website; and including comment and input on M&E findings 
within annual reports and other organisational documents. 

• Recognising that the information gathered is not an end, in itseif, with: forms of 
information and information flows to be linked with decision-making and managerial 
roles and role-players; planning to include time for and focus on reflection; and focus 
to be placed on building capacity and interest not only in gathering and analysing 
information - but in applying insights. 

These learnings and those relating to the steps seen as necessary in establishing an M&E 
system - as outlined below - will be applied in the context of the City’s M&E Framework and 
its rollout. 


3.3.2. Steps viewed as important in setting up and implementing an M&E systemi 

Experience is witness to the fact that implementation of an M&E system is as important as 
its design. Practice has led to the identification of the following steps as key, in ensuring a 
smooth implementation process: 


Table 3.1: Steps to consider in implementing an M&E system or framework 


Step 

Activities 

1 . Establish clarity in 
terms of the M&E 
system’s purpose, 
scope and 
comprehensiveness 

• Ask questions regarding the rationale for establishing an M&E system, 
with due consideration of the needs of various stakeholders, and the 
type of system required e.g. in terms of level of participation, level of 
detail, resources required, etc. 

• Develop a centrally-driven policy to ensure sound implementation of 


^ Inputs drawn from Lahey (2009), Wageningen UR (2010), the Presidency (2010) and Kusek& Rist (2004) 
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Step 

Activities 


the M&E system. 

2. Establish a clear 

view of roles and 
responsibilities in 
relation to the M&E 
system 

• Identify roles and responsibilities with regards both monitoring and 
evaluation-related activities. The key responsibility for monitoring often 
rests with role-players such as the programme or project head. Parties 
accountable for oversight and ensuring quality of monitoring may 
perform an additional support role (e.g. in terms of systems, 
methodology, rigour, etc.). In contrast, use is frequently made of an 
independent party to carry out duties in respect of evaluation, with 
input often gathered from the project or programme head and other 
related sources. 

• Assign key activities associated with the M&E system to various role- 
players - defining factors such as timeframes, frequency of M&E 
activities, stakeholders, deliverables and application of deliverables. 

• Develop and communicate a formal policy document, documenting 
roles and responsibilities - to ensure a shared understanding. 

3. Build capacity for 
design, 

implementation and 
refinement of the 

M&E system - and 
optimal use of 
information 

• Establish a central driver of M&E expertise within the organisation, to 
develop and implement policy, aid implementation, build learnings and 
support capacity development. 

• Actively build M&E capacity, with focus placed on establishing 
monitoring capabilities across the organisation, supported by the 
establishment of a centralised pool of specialist skills to aid in 
activities such as undertaking evaluations, independent of project or 
activity owners. 

• Identify the required competencies for those involved in M&E 
activities, and ensure capacity building - to strengthen the quality of 
M&E activities and to reinforce the system’s credibility. 

• Establish a level of M&E understanding amongst non-technical ‘users’ 
of M&E information. 

4. Focus on identifying 
the right questions 
for M&E, alongside 
information 
requirements, 
indicators and users 
of information - 
linking the vision with 
aligned short, 
medium and long- 
term objectives 

• Identify the long-term vision and build a ‘theory of change’ on how to 
attain it - translating desired impacts into outcomes, outputs, activities 
and inputs (in the short, medium and long term). 

• Align the short, medium and long-term translation of the vision with 
budget and other resources required for delivery - i.e. inputs (e.g. via 
the IDP and Service Delivery and Budget Implementation Plan 
[SDBIP] processes). 

• Identify the appropriate indicators and measurement mechanisms for 
M&E activities in the context of defined activities, outputs, outcomes 
and impacts - with these supported by clear targets and baselines in 
terms of each indicator. 

• Establish an approach through which to communicate the vision and 
its translation into short, medium and long-term plans, so that all are 
able to focus on and carry out M&E activities against appropriate 
indicators. 

• Establish a plan for rolling out M&E activities - with the plan including 
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Activities 


identification of timeframes, role-players, activities, deliverables and 
end-users of information. 

5. Plan for gathering 
and coordination of 
meaningful, clear, 
user-friendly, 
information 

• Establish how M&E information will be gathered and collated. 

• Determine who will be involved in the information gathering and 
collation stage. 

• Identify if additional systems or tools are required for data collection to 
result for sound data collection and data credibility - and if so, 
develop/ establish/ procure. 

• Establish who the stakeholders and direct beneficiaries of data 
collection will be, to ensure data collected is user-friendly, meets 
requirements and will aid reflection and refinement of plans - and 
achievement of outputs, outcomes and impacts. 

• Identify how and where data will be stored - focusing on data 
accessibility, confidentiality and secure storage requirements. 

6. Establish how 
information will be 
analysed, and how 
reflections and 
learnings will be 
represented 

• Identify how information will be analysed. 

• Identify where additional support, methodology, tools or specialists are 
required, to support analysis. 

• Present information and analysis in a palatable form, for engagement, 
reflection, debate and application by stakeholders within and external 
to the organisation. 

• Consider how reflections and analysis will be shared, and 
mechanisms through which to improve this, to enhance wider 
reflection and optimal application of findings. 

7. Ensure quality 
communication, 
feedback and 

stakeholder 
participation - aiding 
implementation of 
improvements, while 
enhancing system 
and organisational 
legitimacy 

• Identify stakeholders and information requirements - in relation to 

M&E activities and deliverables. 

• Ensure communication at regular stages throughout the M&E cycle - 
to build an understanding of activities underway, access to findings, 
stakeholder engagement and participation, commitment to the system, 
and application of learnings. 

• Establish regular opportunities for stakeholder engagement and 
interface in respect of plans, delivery and M&E - building a sense of 
joint purpose and focus. 

• Consider application of findings in respect of plans, budgets, resource 
requirements - and fundamental assumptions and the theory of 
change underpinning these, adapting the appropriate lever as needed. 

8. Implement effective 
change management 
practices - and clear 
leadership - to 
support and embed 
M&E 

• Recognise that establishing an M&E system is a process that 
necessitates ongoing reflection and refinement. Focus is needed on 
building enthusiasm for, and understanding of the need for M&E 
information, across various levels in the organisation - thereby 
matching supply of M&E information with demand. 

• Ensure the leadership team is focused on driving in M&E 
implementation, given that capacity development and ongoing M&E 
requires leadership commitment and consistent application, with 


The City of Johannesburg’s Monitoring and Evaluation Framework 

1 5 I Page 


14 May 2012 


Step 

Activities 


implementation and necessary refinements taking years. 

• Consider implementing incentives (these may be positive or negative) 
to drive interest in - and use of - M&E information. 

• Attend to the organisational culture, focusing on building a results- 
oriented, evaluation-focused organisational culture - a necessary 
requirement if M&E principles are to remain in place in the long term. 


Learnings gained from M&E practice in other contexts is invaluable in ‘getting it right’. As 
noted, the above learnings are amongst many that have been taken into consideration in the 
development of this M&E Framework. 


3.4. Driving outcomes across South Africa: The nationai approach to M&E 

M&E systems have been implemented for many years in a range of environments, with 
increasing focus on the importance of an ‘outcomes’ or ‘results based’ approach. Building on 
the foundation offered by practice implemented within other settings. South Africa recently 
chose to adopt an outcomes approach to M&E, using this as the mechanism through which 
to ensure the rollout of delivery objectives on a national scale. 

The DPME, in its role as driver of M&E from a national perspective, has focused effort on 
establishing a common approach to M&E amongst all delivery agents across the three 
spheres of government. The department has built on the Presidency’s 2010 ‘Guide to the 
Outcomes Approach’, which sets out government’s approach to M&E and the management 
of each of the 12 outcomes that, if achieved, would collectively address government’s 10 
strategic priorities for 2014. The recently published NEPF provides a clear framework for 
implementing evaluation activities - and serves as a valuable reference point in ensuring 
consistency of approach, while also allowing individual departments to customise the system 
to suit their needs. 


3.5. The ‘outcomes approach’: A foundation for meaningful M&E 

The outcomes approach “is designed to ensure that government is focused on achieving the 
expected real improvements in the life of all South Africans... [it] clarifies what we expect to 
achieve, how we expect to achieve it and how we will know whether we are achieving it” 
(The Presidency, 2010, pp.9-10). In introducing this approach, the Presidency notes that 
“[ojutcomes refer to a changed state of being... They describe the effects, benefits or 
consequences that occur due to the outputs or programs, processes or activities.” (2010, 
P-12). 

The Presidency (2010, pp.1 1-12) notes the following as the central elements included in the 
outcomes approach: 

• Inputs: “...everything we need to accomplish a task. This could be in terms of 
finance, human resources, infrastructure etc.” 
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• Activities: “...a collection of functions (actions, jobs, tasks) that consume inputs and 
deliver benefits or impacts.” 

• Outputs: These “can be immediate and intermediate... direct products and services 
generated through processes or activities without specific reference to their ultimate 
purpose.” 

• Outcomes: “...a changed state of being... They describe the effects, benefits or 
consequences that occur due to the outputs or programs, processes or activities. The 
realisation of the outcome has a time factor and can be in either the medium or long- 
term.” 

• Impacts: The long-term “developmental result at a societal level that is the logical 
consequence of achieving specific outcomes” 

The figure below depicts the relationship between these various elements diagrammatically. 


Figure 3.1: Elements within the outcomes approach (Adapted from: The Presidency, 

2010, p.11) 



In terms of the outcomes approach, the Presidency has established a number of 
mechanisms through which to ensure rollout of the system, and achievement of the 12 
outcomes. These are as follows: 

• At a national level. Performance Agreements signed by the President and each 
Minister form the key vehicle through which accountability for delivery in respect of 
the 12 defined outcomes is assigned to the various ministries. 

• Each Minister is then required to sign an Administrative Performance Agreement 
with his or her Director General (DG) - with focus placed on alignment with the 
primary Performance Agreements. Through this process. Administrative Performance 
Agreements serve as a mechanism through which delivery objectives are cascaded. 
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• Outcomes are also cascaded to provincial heads, through Intergovernmental 
Protocols signed between the President and each Premier. 

• In addition to the above mechanisms, commitments to delivery are concretised 
through the development of a Delivery Agreement per outcome - with signatories 
including all those whose involvement is necessary for outcome achievement. This 
includes role-players from across the spheres and sectors of government - and may 
include role-players who are external to government, where appropriate. These 
agreements are negotiated, specific in nature, and contextualised. 

The figure below depicts the various forms of interface between these different elements 


Figure 3.2: Putting the outcomes into practice - different levels of agreements 



At a local government level, the primary outcome of relevance is Outcome 9, articulated 
through the vision of a “responsive, accountable, effective and efficient local government 
system”. In practical terms, this outcome has been committed to by means of a Performance 
Agreement signed between the President and the Minister for Cooperative Governance and 
Traditional Affairs, reflected further in an Administrative Performance Agreement signed 
between the Minister and the department’s DG. Outcome 9 is also committed to by means of 
an Intergovernmental Protocol signed between the President and the Premier of Gauteng. 
Lastly, it is committed to within a Delivery Agreement, focused on engaging all parties 
involved in its achievement - including the City of Johannesburg. 
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In considering the outcomes approach - as represented in Figure 3.1 - it is the flow of 
elements from inputs, activities and outputs, through to outcomes and impacts that is critical. 
This represents the ‘theory of change’ - a ‘logic model’ “...that describes a process of 
planned change, from the assumptions that guide its design, the planned outputs and 
outcomes to the long-term impacts it seeks to achieve” (DPME, 2011, p.20). It provides a 
view of the causal links between the desired long-term impact or goal and each of the 
outcomes identified as necessary for this impact to be realised. Outcomes are then broken 
down further into the associated outputs, activities and inputs. 


3.5.1. The ‘theory of change’: Defining assumptions within the outcomes approach 

A ‘theory of change’ is therefore a reflection of the end goal or impact desired, and the 
outcomes, outputs, activities and inputs viewed as necessary for this end goal to be 
achieved. A set of assumptions underpin identification of each of the elements in the chain. 
Assumptions may arise from experience, facts, insights, formal learnings, research or other 
sources. Through ongoing M&E activities, these assumptions may be surfaced, challenged 
and refined, thereby allowing those using the M&E Framework to apply insights from past 
practice when identifying the most appropriate set of activities, outputs and outcomes 
through which to drive the desired long-term goals (The Presidency, 2010). 

Figure 3.3 below provides a representation of the theory of change - as a chain of influence 
of possible inputs, linked to activities, outputs, outcomes and impacts, in the context of the 
City’s GDS. When considering the planning and budgeting mechanisms, it is important to 
note that ‘outcomes’ articulated in the GDS may not be realised within the short term cycle 
as defined in the annual IDP or even within the five-year timeframe (i.e. within the five-year 
IDP) . Instead, the timeframes associated with some of the outputs and outcomes will only 
be realised in the medium- to long-term IDP (e.g. the IDP for the 2016/21, or 2021/26 
period). Therefore, in the planning and budgeting cycles, there is a need for prioritisation - 
whilst ensuring adequate balance in the delivery agenda. 
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Figure 3.3: Cascading impacts into time-reiated outcomes, outputs, activities and 
inputs 


^ o 




y ^ y 



To identify each of the elements within the theory of change, a process of ‘working 
backwards’ is required - identifying the intended impact and outcomes, the deliverables or 
outputs that will result in these, the necessary activities, and inputs required for these 
activities to be carried out. 

Establishing a sound ‘theory of change’ is heavily dependent on first understanding the 
status quo - and the ‘problem’ or circumstances that interventions, activities or outputs are 
intended to address. Identifying the gap between the status quo and the desired future 
assists in determining which outcomes and supporting outputs and activities are necessary, 
to bridge the distance between the ‘as is’ and the ‘to be’. AusAID notes the benefit of using 
the ‘problem tree’ tool in this regard - portraying ideas relating to underlying constraints, 
cause and effect. This graphic depiction allows for assumptions relating to the current reality, 
necessary conditions for change and possible solutions for the achievement of the long-term 
vision to be clarified, challenged and refined. In this way, a more effective theory of change 
may be established. 

An example of a problem tree - developed in the context of national budget execution - is 
included below. It presents a set of secondary and primary causes (the underlying roots), 
which collectively contribute to the problem (the trunk of the tree - in this case: “low levels of 
budget execution”) and the immediate effect (the branches). Using an analysis tool such as 
the problem tree approach to establish an improved understanding of cause and effect may 
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assist an organisation such as the City to identify all the factors contributing to a problem or 
challenge (Evaluation Toolbox, 2010). Through this, it may be possible to enhance solution 
development - articulated within a refined theory of change, and aligned plans of action. 

Figure 3.4: Example of a problem tree, strengthening analysis (AusAID, 2005, p. 6) 



3.5.2. How does the theory of change align with levels of performance management? 

Are all the elements included within the outcomes approach (and depicted in a chain of 
causality, as part of a theory of change) equally applicable to individual performance as they 
are to group performance? At an organisational level, delivery and the aligned measurement 
and monitoring thereof is focused primarily on outcomes and outputs. Activities and inputs 
then serve as a foundation for ensuring and reviewing delivery by those responsible for 
carrying out individual duties, in support of the defined outputs and outcomes. Further detail 
in this regard - in the context of the City - is provided in Section 3.6 below. 


3.5.3. The theory of change applied - an example 

The figure below demonstrates the process of cascading the concepts relating to the theory 
of change - in the context of a simple example. The diagrammatical representation serves 
as a meaningful approach through which to present the espoused ‘theory’ - with its depiction 
of the outcomes approach and each of the levels of which it is constituted. Through making 
the assumptions that underpin the identified actions clear, by means of a graphical depiction 
such as the one below, these assumptions and the overarching logic are open to 
interrogation, testing, review - and improvement. 

As reflected in Figure 3.3 above, it should be noted that there may be numerous outputs 
contributing to a single outcome, and likewise, there may be numerous activities contributing 
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to a single output - with the importance of accurate cascading evident. This principle may 
apply to any level of the model. 


Figure 3.5: Representation of the outcomes approach through a project exampie 





X 




A well-considered theory of change, active learning and the ability to demonstrate 
organisational agility is essential for success in any setting. When plans are implemented, 
outputs or outcomes that differ from those first envisaged may emerge. Circumstances may 
shift. New challenges may arise. Ongoing monitoring and regular evaluation provides an 
opportunity to identify the approaches that work and those that do not. This process of 
reflection may result in a change to the medium or long-term strategy, alongside short-term 
actions. 

Given the above, it is acknowledged that while the vision may remain the same, the strategy 
(and with it, the identified impacts, outcomes and outputs) may shift - based on learnings 
and insights gained through M&E activities and the observed results. Through reflection, 
analysis of assumptions versus experiences, and fine-tuning of the strategy and plan, M&E 
activities will support the achievement of the City’s long-term vision. Without being open to 
questioning the defined strategy and the associated activities, outputs, outcomes and 
impacts through periodic evaluation and ongoing reflection, the City runs the risk faced by so 
many organisations: following the wrong path. The outcomes approach to M&E provides the 
platform for this process - and the foundation for making assumptions and theories of 
change explicit and available for review, debate and refinement over time. This allows for 
deep, ‘double-loop’ learning - a concept explored below. 
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Box 2.3: Embedding real learning - the art of deep reflection 


Single-loop learning in an organisational context involves the identification and correction of 
an error, in the context of unchanged assumptions, goals, plans or values (i.e. a focus on 
operationalizing these governing elements, rather than questioning them). In contrast, 
double-loop learning involves a process of also questioning underlying assumptions. The 
latter form of learning may result in a change to the underlying assumptions, goals and 
principles themselves - shifting the manner in which organisational strategies are defined 
(Argyris and Schon, 1978, cited in Smith, 2001). This is the foundation of Senge’s approach 
to the learning organisation - where underlying mental and views are put to the test, to 
enable ongoing learning and change. 

“Single-loop learning seems to be present when goals, values, frameworks and, to a 
significant extent, strategies are taken for granted. The emphasis is on ‘techniques and 
making techniques more efficient’ (Usher and Bryant: 1989: 87). Any reflection is directed 
toward making the strategy more effective. Double-loop learning, in contrast, ‘involves 
questioning the role of the framing and learning systems which underlie actual goals and 
strategies (op. cit.).” (Smith, 2001, p.4) 


3.6. Ensuring alignment with the national perspective 

As noted in the Joburg 2040 GDS, the City has chosen to align itself with the outcomes 
approach adopted by national government, in defining its envisaged long-term future. While 
the long-term vision for the City has been reflected through a set of outcomes, delivery of 
these outcomes requires systematic focus and the support of a sound and commonly 
understood M&E approach. Recognising its role as a key link in the delivery chain of 
government, the City has chosen, through the M&E Framework and the tools that will 
support it, to establish a rigorous approach to M&E - in support of its commitment to 
Outcome 9 and the vision reflected in the GDS. 

The diagram below depicts the interface between national, provincial and local government 
commitments - in the context of the outcomes approach. Ideally, a two-way interface should 
exist between the various strategies and policies, with delivery, experiences, challenges and 
data collected within each sphere integrated into the collective delivery framework of 
government. Also reflected is the manner in which the City’s vision aligns with national and 
provincial planning priorities - and the planning mechanisms used to drive delivery. 
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Figure 3.6: Alignment between national/ provincial priorities, and City outcomes 
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In delivering on the principle of alignment, the City has identified a set of outcomes for 2040 
that are aligned to the national outcomes, the provincial priorities and Programmes of Action, 
the City’s vision for 2040 and the principles and paradigms underpinning the Joburg 2040 
GDS. Each long-term outcome defined in the GDS has been broken down further into a set 
of outputs. 

Associated activities for delivery in relation to the long-term, medium-term and short-term 
outcomes and outputs have been identified through a set of five-year I DPs (reflected above 
as ‘cluster priorities and programmes’). The five-year Mayoral Flagship Programmes define 
key outcome based programmes to be implemented for the Mayoral term (which serve to 
operationalize the GDS). These are further translated into five-year cluster-specific priorities 
and plans that address integrated delivery within each of the four clusters. All these plans 
are then annualised within an annual IDP, the SDBIP, business plans and individual 
performance plans. 

With effective M&E, the mechanisms depicted above and in the figure below are able to 
interface with each other, forming a system that allows for active learning based on data 
collection, analysis, reflection, improved ‘theories of change’ that are used to underpin plans 
and programmes, and greater delivery-related adaptation. 
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Figure 3.7: Alignment between levels of achievement across the outcomes approach - 
and the City’s planning mechanisms 
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Building on the outcomes approach depicted above, the City has adopted a further level of 
distinction within its M&E Framework, in relation to the concept of ‘outputs’, and those 
activities that contribute to their achievement. 

Ongoing delivery on daily priorities is as important as achievement of the big ideas - a truth 
acknowledged by other organisations^ that have chosen to adopt a more nuanced approach 
to the identification of planned outputs. Given the need for a balanced focus on short, 
medium and long-term responsibilities, the City’s M&E Framework allows for a balance in 
emphasis across three categories of delivery - namely, ‘continuous improvements’, 
‘impactful improvements’, and ‘broad initiatives’. 

These concepts emerge as a consequence of the recognition that focusing only on outputs 
directly associated with those outcomes included in the long-term strategy may lead to the 
neglect of the City’s day-to-day service delivery obligations. Certain outputs that are 
necessary for short and medium-term delivery may also not align directly to the ‘big’ ideas 
reflected in the long-term GDS. Pursuing long-term ideas alone would be to the detriment of 
those responsibilities that fall squarely within the City’s core mandate - e.g. waste collection 
and road maintenance. Likewise, if we attended to our day-to-day responsibilities only, the 
long-term vision reflected in the 2040 GDS will remain a dream. Long-term initiatives such as 
large capital-intensive infrastructure projects frequently need investments and active 
development at least ten to fifteen years before project completion. This necessitates a 
visionary focus - and action in the short and medium-term, for long-term success. 

The figure below provides a representation and definition of these concepts, within the 
context of the City’s long-term planning horizon. 


^ Including, for example, the South African Revenue Service 
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Figure 3.8: Categorisation of focus areas - for baianced prioritisation 


Dealing with the basics (Continuous improvements) 

Improvements that are ongoing in nature, focused on day to day delivery requirements 
E.g. Addressing unfilled pot-holes on a daily basis- while improving materials and techniques used, to 

reduce maintenance needs 


Impactful improvements 

Improvements that hold significant impact, and may be delivered within a defined Term of Office 

E.g. Flxingthe City’s billing system 


Broad initiatives 

Initiatives that are more long-term in nature, aligned to the 2040 GDS- and may be more complex and 
intensive (eg. in terms of capital or technical requirements) 

E.g. Eliminating waste to landfill 


I 2011 -2015 I 2016-2020 I 2021 -2015 | 2026-2030 I 2031 -2035 | 2036 - 2040 


Maintaining a balance between different types of outputs is a significant challenge for all 
organisations, particularly where resources are limited and where prioritisation is necessary. 
The City has adopted these categories in acknowledgement of the fact that ongoing delivery 
is as important as the achievement of big ideas. Balanced focus is required in respect of 
activities and outputs focused on all three types of outputs. 

The above distinctions and the broader application of the M&E Framework within the |City 
takes place in the context of a tightly regulated and managed annual and five-year planning 
schedule. The diagram below presents an overarching view of the application of M&E in this 
context. A view is presented of the various forms of planning, monitoring, evaluation and 
reporting in place within the City, alongside associated timeframes. Each of the processes 
and reports linked to planning, delivery and M&E is dependent on the others in the cycle, 
with the importance of accurate cascading of goals, indicators and measures evident. 

Also represented below are the various forms of reporting associated with the different M&E 
periods. Monitoring reports such as the quarterly review report - as developed in respect of 
the business plan and individual scorecard implementation - aid in building an 
understanding of progress and ensuring ongoing strategy-aligned implementation. In 
contrast, evaluation reports, such as the Mid-Term Performance Assessment Report 
(undertaken in respect of the five-year IDP and cluster plans) represent time-specific 
analyses of commitments delivered, and those that remain unmet. As such, the City’s 
annual, five-year and longer-term planning cycle effectively includes elements of a 
predefined M&E plan. This is depicted alongside the annual performance planning, 
monitoring, evaluating and reporting cycle. 

All activities also take place within a broader national domain, with the objectives, direction 
and M&E practices (e.g. indicators and targets used) from other spheres of government 
informing the City’s approach. 
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Figure 3.9: An overview of pianning, monitoring, evaiuating and reporting in the City 


M&E Framework 
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Data-driven; Aligned with national and provincial system? focused on both 
administrative and political arms. 









4. M&E within the context of the City’s pianning mechanisms 


The chapter that follows provides clarity on the principles and values underpinning the City’s 
M&E Framework, before presenting an overview of the delivery context within which the 
Framework operates - i.e. the Joburg 2040 GDS, with its vision, priorities and defined 
outcomes, the successive five-year IDPs through which the GDS will be realised, and the 
annual IDPs. Focus is placed on clarifying how these various planning tools align with the 
elements included in the outcomes approach. Chapter 5 then builds on these points of 
clarity, detailing the practical steps through which the various M&E elements are put into 
action. 


4.1 . Guiding principles for the City’s M&E Framework 

If we are to realise our long-term strategy as reflected in the GDS, all role-players need to 
work together. This is the case for both delivery-related actions and those activities more 
specifically focused on ensuring sound M&E, in support of intended areas of impact, and the 
associated outcomes, outputs and activities. M&E approaches enable us to track progress 
against plans and defined objectives, making appropriate adjustments as necessary. To 
focus all members of the City’s ‘team’ on a common approach to M&E, a set of guiding 
principles and values have been adopted - with these underpinning all M&E activities. The 
principles adopted in respect of M&E are as follows: 

Box 4.1: Principles guiding the City’s approach to M&E (GWMES, cited in DPME, 2011, 
P-3) 

M&E: 

1 . "... should be development-oriented and should address key development priorities 

of government and of citizens. 

2. ... should be undertaken ethically and with integrity. 

3. ... should be utilisation-oriented. 

4. ... methods should be sound. 

5. ... should advance government’s transparency and accountability. 

6. ... must be undertaken in a way which is inclusive and participatory. 

7. ... must promote learning.” 


Implementation of each of the above principles in the City’s context will take place through: 

• Ensuring M&E activities focus on measuring progress towards poverty-reduction, 
economic inclusivity and other indicators of improved development - as per the 
principles defined within the GDS. The City will carry out regular monitoring on levels 
of deprivation and inclusivity, using impact assessments to evaluate development- 
oriented initiatives. 

• Data collection, analysis and reporting that is conducted in an honest, respect-driven 
and unbiased manned, supported by sound M&E systems and capable internal and 
external role-players. 
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• A focus on presenting M&E information and findings to the City’s internal and 
external stakeholders in a manner that is accessible, user-friendly, applicable in 
driving service delivery improvements - and able to contribute to the knowledge-base 
of the City. 

• The use of M&E methodology that is customised to and appropriate for the identified 
inputs, activities, outputs, outcomes and impacts - with triangulation (i.e. employing 
multiple sources to improve reliability and validity) and the use of appropriate 
indicators aiding in building credible findings. 

• Continued focus on ensuring optimal stakeholder participation and an environment 
for transparent engagement, recognising the value this holds for the refinement and 
revision of the City’s strategy and plans - and ultimately, the growth of an active, 
responsible citizenry. Ongoing M&E activities also serve to support accountability, 
with this located at two levels: the accountability held by those receiving information 
through the M&E system, and the accountability held by those providing input and 
feedback into the M&E system. Those receiving information from the M&E system 
are obligated to drive reflection and improved performance following the receipt of 
M&E feedback, while those who are able to provide feedback to the City through its 
M&E system are also obligated to provide honest, open input - recognising the 
benefit of this for improved delivery. 

• Genuine, focused interactions with all of the City’s stakeholders - regardless of who 
they are or where they are from - across all M&E activities, in the interests of building 
“a vibrant, equitable African city, strengthened through its diversity’ (CoJ, 201 1 , p.3). 

• Systems and processes which support reflection, learning and an improved 
understanding of causality - with regular M&E activities leading to refined plans and 
activities, and a process of organisational ‘renewal’. Opportunities for individual, team 
and the organisational learning will be maximised through M&E - with regular 
reflection on the roll-out of plans enabling active learning, greater agility and, with the 
application of insights emerging, improved outcomes. 


4.2. Values framing the City’s deiivery approach 

The City’s Value Code, identified as applicable in the context of the City’s performance 
management system, is also included here, given the applicability of the values to the M&E 
Framework that in itself serves to support effective performance management. The section 
that follows provides an overview of the manner in which these values translate to the 
domain of M&E. 
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Box 4.2: Values guiding the City’s approach to M&E (CoJ, 2009a, pp.15-16) 


“The values are the heart and soul of the City. They are the fundamental notions of ideal 

behaviour - the set of beliefs upon which decisions are made and actions taken. The 

performance management related principles drive the design and implementation of the 

performance management system itself. 

The City of Johannesburg’s Value Code: 

We: 

• Are accountable to the community that we serve and act on behalf of the residents of the 
City of Johannesburg by focusing our efforts on discovering and meeting our customers’ 
needs; 

• Believe in transparency and are open to scrutiny in all that we do; 

• Are honest and uncompromising in producing work that meets the service delivery 
requirements of our City; 

• Are committed to the transformation of our society and our workplace and will ensure 
equity and anti-discrimination in all we do; 

• Show concern for people, support and respect our colleagues and customers and 
encourage the growth and development of everyone we work with; 

• Value the fair and consistent treatment of staff and customers; 

• Believe in transparent participation promoting fair, yet efficient, decision-making while 
encouraging individuals to accept responsibility for their work and contribution to the 
City.” 


Implementation of the above values in the City’s context will take place through the following 
M&E associated actions, amongst others: 

• Ongoing engagement with citizens, through processes such as the GDS Outreach 
and other forms of active, two-way communication; 

• Honest communication of M&E plans, processes, delivery, findings and impact; 

• Rigorous engagement in discussion and analysis through the various M&E platforms 
- to ensure delivery is aligned across the short, medium and long-term; 

• Driving the concept of transformation and open engagement through both processes 
and deliverables associated with M&E activities; 

• Interacting openly and constructively with all across the City, in all M&E interactions - 
and thereby promoting growth and improvement at the individual, unit and 
organisational level; 

• Ensuring all understand their areas of responsibility, and are equipped to deliver, 
through clarity provided in respect of M&E roles and responsibilities, and the 
provision of supporting methodology and tools; and 

• Engagement with all internal and external stakeholders in a responsible, fair, 
transparent and accountable manner - recognising that the vision and outcomes 
stated in the City’s long-term GDS depend on collective and integrated delivery by 
all. 
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The Joburg 2040 GDS serves as the foundation on which M&E activities are centred, given 
its articulation of outcomes and impacts to be realised in the long-term. This therefore serves 
as the starting point for understanding the emphasis of M&E in the City. 


4.3. The Joburg 2040 GDS: the ultimate compass for M&E in the City 

The 201 1 local government elections saw the City enter a new Mayoral term, ushering in a 
refinement process in terms of the City’s long-term vision and guiding framework at the time, 
the ‘2006 GDS’. The City undertook an intensive review of City priorities, building on existing 
strategies and plans, commitments and areas of delivery - while acknowledging new 
challenges and priorities emerging at other levels (international, regional, national, provincial 
and local). An extended research, analysis, reflection and stakeholder engagement process 
culminated in the launch of the City’s Joburg 2040 GDS in October 201 1 - a long-term 
strategy defined as the ‘compass’ by which the City and all those who work within it will steer 
themselves, to achieve a commonly held vision: 

“Johannesburg - a World Class African City of the Future - a vibrant, equitable African 
city, strengthened through Its diversity; a city that provides real quality of life; a city that 
provides sustainability for all Its citizens; a resilient and adaptive society. ” (CoJ, 2011, p.3) 

The City aims to deliver. To do so, continuous monitoring and regular evaluations are 
necessary, to ensure all actions and outputs are on track and aligned with the envisaged 
outcomes, and, ultimately, the desired impacts reflected within the GDS. 

But M&E activities are not only important in ensuring the ongoing delivery on the outcomes 
reflected within the GDS. The concept of governance is central to the GDS, with its role - 
and the importance of M&E activities for its achievement - addressed below. 

4.3.1. The GDS paradigm, and the location of ‘good governance’ within it 

To focus attention, the Joburg 2040 GDS incorporates a ‘GDS paradigm’, which serves as a 
lens through which to view and understand the City’s responsibilities, priorities and the 
outcomes envisaged for the future. The paradigm sets the City’s sights on the concepts of 
resilience, sustainability and liveability, while acknowledging that the achievement of these 
concepts will require careful management of the tensions between four primary inter-related 
drivers: human and social development; inclusive and productive economic growth; 
environment and services; and governance. These interrelationships are depicted in the 
figure below: 
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Figure 4.1: The Joburg 2040 GDS paradigm (CoJ, 201 1, p.23) 


Human and 
Social Development 


Resilience • Sustainability - Liveability 


Economic 

Growth 


Governance 


Environment and Services 


This paradigm acknowledges ‘governance’ as a critical driver. The Joburg 2040 GDS reflects 
further on the importance of “[sjustained, regular and non-partisan participation in city- 
development - by all types of City stakeholders”, communication, trustworthiness, 
accountability, responsiveness - and a clear, transparent governance framework through 
which to mediate the City’s tensions between “urban management, enforcement and the 
protection of rights” (CoJ, 2011, p.32). Ensuring a balanced delivery on commitments, while 
maintaining an ongoing awareness of possible tensions between different priorities so that 
these can optimally managed, requires up to date information and the type of insights that 
can only emerge when a rigorous M&E approach is in use. The concepts relating to M&E are 
located firmly within the same domain as ‘governance’, with governance requiring careful 
implementation of appropriate structures, systems, processes, people and the right type of 
culture to support the principles of fairness, accountability, responsibility and transparency 
are applied by all. 

Applying the M&E Framework requires an understanding not only of the activities and 
deliverables associated with the framework, but an appreciation of the principles and long- 
term outcomes the framework ultimately supports. These are detailed below. 

4.3.2. Defining the City’s future focus: The GDS’ principies and iong-term outcomes 

Building on the direction provided by the GDS paradigm, the City has defined six guiding 
principles that provide clarity on its approach to development - aiding decision-making, 
prioritisation and budgeting. The figure that follows depicts these principles, alongside the 
vision identified for the Johannesburg of 2040, the key drivers, and four emerging outcomes. 
The latter serve as the basis for planning and navigation in the short, medium and long-term. 
Within the GDS, these outcomes are reflected through a set of outputs that are defined in 
relation to each outcome. Proposed indicators are also detailed, providing an initial view of 
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the tools through which achievement of the envisaged outputs and outcomes can be 
assessed. 

All plans and deliverables across the City should ultimately focus on steps that enable 
achievement of these outcomes and thereby, the Joburg 2040 Vision. Aligned outputs assist 
in driving delivery, as reflected through the identified indicators included in the GDS itself. 


Figure 4.2: The Joburg 2040 GDS vision, paradigm, principies and outcomes 



Delivery on the four outcomes is dependent on a concerted effort and focus by all within the 
City - particularly as long-term outcomes frequently require the achievement of targeted 
outputs and supporting activities in the short and medium-term. 

What does this mean for M&E practice in the City? The figure below provides an indication 
of the applicable level of emphasis within the outcomes approach, in relation to the GDS- 
focused goals. As is evident, M&E activities undertaken in the context of the GDS’ ultimate 
goals should emphasise delivery on impacts and to an extent, outcomes. Each mid-term 
review undertaken in respect of delivery on the GDS will need to reflect on these elements, 
within the outcomes approach (please see Figure 3.8, should further clarity be required). 

This targeted focus is depicted in the figure below. 
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Figure 4.3: Focus ofM&E activities in the context of the City’s iong-term GDS 


Levels of Performance Indicators Planning Levels 



4.4. Locating M&E activities associated with the five-year IDP and cluster planning 
process 

As noted in Chapter 3, the legislated five-year IDP process assists in translating the GDS’ 
long-term impacts and outcomes into measurable and achievable targets. In the City’s case, 
these are framed within the context of the Mayoral Flagship Programmes, located within five- 
year IDPs that contain cluster-specific priorities and plans. It is through only through working 
‘backwards’ from 2040 to our current IDP period that we will be able to successfully cascade 
the intended impacts and outcomes reflected in the GDS, and ensure progressive delivery 
through each five-year period. This is reflected in the figure below. 


Figure 4.4: Focus of M&E activities in the context of the City’s iong-term GDS 
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Through this process of cascading, the City’s M&E mechanisms across each successive 
period serve as a form of glue, aligning the delivery targets of all sections of the City through 
a metro-wide ‘theory of change’, and thereby the supporting achievement of City-wide 
outcomes and impacts defined within the GDS. While the City has performed certain M&E 
functions to date, the GDS lays the foundation for a refined M&E Framework. 

The figure below provides a clear view of the focus of the five-year I DP and associated 
cluster plans, where M&E initiatives are more oriented towards understanding the 
achievement of medium-term outcomes and in some areas, outputs, aligned to the long-term 
impacts. 


Figure 4.5: Focus of M&E activities in the context of the City’s long-term GDS 
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4.5. Locating M&E activities in relation to the annual IDP, SDBIP, Business Plans and 
Scorecards 

The final level of M&E activity within the City relates to the more short-term elements in the 
City’s planning cycle - reflected through planning tools such as the SDBIP, the annual IDP, 
annual Business Plans and individual scorecards. This is depicted in the figure below. 
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Figure 4.6: Focus ofM&E activities in the context of the City’s annuai pianning toois 


Levels of Performance Indicators Planning Levels 



Through rigorously undertaking each level of planning outlined above, and ensuring that the 
associated delivery, M&E, reporting, communication and feedback mechanisms are in place, 
the City will be well-positioned to deliver on its envisaged long-term priorities and ideals - as 
reflected in the GDS. This clearly requires effort from all parts of the organisation, and each 
individual who contributes on a daily basis in ensuring delivery to those we aim to serve. 

Without following a systematic process of cascading impacts, outcomes, ad outputs - such 
as that detailed above - any achievement of our long-term goals will be a case of ‘hit or 
miss’. Focused cascading of rather than a carefully orchestrated performance of which all 
can be proud. 

The following chapter addresses the ‘how’ of M&E, providing further clarity in the context of 
five key levels of focus in respect of delivery - whether related to impacts, outcomes, 
outputs, activities or inputs: 

• Planning for M&E; 

• Conducting monitoring; 

• Conducting evaluations; 

• Reporting on delivery; 

• Communication and feedback. 
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5. HOW: Putting the M&E Framework into practice 


The chapter that follows addresses the question of how the M&E Framework should be 
implemented in practice. It therefore seeks to integrate the ideas reflected in the preceding 
sections, with focus placed on the establishment of a City-specific M&E Framework. As 
noted, on final adoption of this overarching framework, a practical guideline for step-by-step 
implementation will be provided, in the form of a supporting M&E Handbook. 

Emphasis here is placed on the following five key phases within the M&E rollout process: 


• Phase 1 : 

• Phase 2: 

• Phase 3: 

• Phase 4: 


• Phase 5: 


Planning for M&E; 

Conducting monitoring; 

Conducting evaluations; 

Reporting on M&E findings - within the context of the City’s reporting 
mechanisms (e.g. the annual report, external assessments, surveys, 
etc.) and any alternative reporting tools or mechanisms deemed 
appropriate; and 

Communicating and providing feedback in respect of M&E and 
delivery. 


Details relating to the roles and responsibilities associated with the M&E Framework are 
included in Annexure 2. Certain areas of responsibility are also commented on within this 
chapter, in the context of specific activities. 


Before addressing each of the phases highlighted above, focus is placed on the place M&E 
activities occupy in the context of the City’s planning cycle. This is then followed with a 
description of each of the five phases highlighted above. 


5.1 . Placing M&E activities within the City’s planning cycle 

Figure 5.1 below depicts the relationship between M&E planning and the typical processes 
and steps undertaken in planning for and delivering on a departmental, organisational or 
project plan. The alignment between project/ departmental plan-specific activities and M&E 
is reflected in the context of four elements: planning, acting, monitoring and evaluation. 
While not reflected here, it should be noted that the steps of ‘reporting’ and ‘communicating’ 
M&E analysis are also critical, if an organisation is trying to truly maximise the benefits that 
M&E processes and systems could bring. 

An organisation-wide M&E system will only succeed if the steps reflected on here are carried 
out at multiple levels across the organisation - i.e. at a City-wide, cluster, departmental/ 
entity, and individual level. Planning must include consideration of M&E activities at all 
levels, for delivery in the context of the ‘whole’. 


The City of Johannesburg’s Monitoring and Evaluation Framework 

37 I Page 


14 May 2012 



Figure 5.1: The M&E cycle in the context of an organisational planning/ project cycle 
(Adapted from Wageningen UR, 2010) 



The process of planning for M&E forms the first step of the cycle, with the need for dual 
planning in respect of delivery (at various levels - e.g. organisational and cluster-related) 
and the associated M&E activities reflected above. Planning for M&E activities and 
deliverables should therefore occur at the same time that strategic planning or project 
planning activities are undertaken - i.e. at the very start of the performance cycle. 

M&E activities cannot be considered independently of the overall strategy or project within 
which they occur. Through identifying the approach to be followed in relation to M&E, 
questions may be raised about the project or strategy itself. In this way, M&E supports 
further learning and the refinement of action plans. This is reflected in the figure below, in the 
context of various ‘loops’ of learning that flow between the M&E system, and plans and 
processes for operational delivery. 
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Figure 5.2: M&E in the context of the project strategy and operational rollout (Adapted 
from IFAD, 2002a, p.4) 
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Two levels of planning are therefore necessary in the context of an outcomes approach to 
M&E. The first relates to the organisational, cluster, departmental or project strategy/ plan, 
while the second relates to M&E. While the former is critical in ensuring we have our sights 
set on the right goals, the latter supports delivery, keeping actions on track and allowing for 
corrective action where necessary. M&E planning therefore serves as a necessary 
foundation for meaningful tracking, assessment and review of performance and delivery. It is 
this area of planning that forms the primary focus of this document. Given the fact that M&E 
activities are of little use when carried out in the context of a poorly devised organisational, 
cluster, departmental or project plan, an overview of operational planning and the steps 
involved is provided in Annexure 3. The issues flagged within the annexure will also be 
included within subsequent revisions to the City’s Performance Management Framework. 
The steps specifically associated with planning for M&E are outlined below, with these 
followed by detail relating to the other key elements of the M&E cycle (i.e. monitoring, 
evaluation, reporting, communication and feedback). 


5.2. Phase 1 : Planning for M&E in the City 

Figure 5.3 below provides a view of the key steps that are to be addressed when planning 
for M&E. This is placed in the context of other forms of planning - e.g. organisation-wide 
planning. The four questions framed in relation to M&E planning form the basis for the step- 
by-step process that is addressed in the section that follows. 
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Figure 5.3: Steps associated with M&E Pianning 
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Integrating aii into a City-wide and Ciuster-specific M&E Pians 




















5.2.1. ‘How will we know we have achieved our plans?’ Identifying indicators (Step 1) 

While the delivery or operational plan itself should address questions relating to what the 
City aims to deliver (whether in the short, medium or long-term]), how will we know we are 
on track - or that we have achieved our original objective? Regardless of whether we are 
looking at the achievement of impacts, outcomes or outputs as defined in the operational 
plan, we need to identify the appropriate indicator that will enable assessment. It is only 
through measuring the right things, and assessing delivery in the appropriate way, that the 
City will be able to ensure sound delivery across all plans, adapting when necessary. 

The first step in preparing for M&E relates to the identification of indicators. These “...are 
measurable or tangible signs that something has been done or that something has been 
achieved” (Shapiro, 2002, p.14). They are “the quantitative or qualitative variables that 
provide a simple and reliable means to measure achievement, to reflect the changes 
connected to an intervention, or to help assess the performance of an organisation against 
the stated outcome” (Kusek & Rist, 2004, p.65). 

Adopting the outcomes approach to M&E necessitates the identification of indicators, targets 
and baselines for each aspect of the model, as depicted in the figure below. For example, an 
outcomes indicator such as “percentage of households with access to basic water” may be 
identified alongside a more short-term activity indicator such as “number of inspectors 
trained”. This allows progress to be monitored and adjustments to be made at the 
appropriate level (i.e. in respect of inputs, activities, outputs, outcomes and goals or impacts) 
- within the context of the relevant planning mechanism (e.g. business plans, the annual 
IDP, the SDBIP, the Cluster Plan, or the GDS). 

Figure 5.4: Identifying indicators in the context of the outcomes approach to M&E 


Levels of Performance Indicators 


Impacts (Goals) 


Outcomes 



Outputs 


l=> 


Planning Levels 


Joburg 2040 GDS 


GDS/ 5-year IDP/ 5-year Cluster 
Plans 


SDBIP 

Annual IDP and Cluster Plans 
Annual Business Plans 
Scorecards 


When selecting indicators, focus should be placed on ensuring alignment with: 
• The interests and concerns of various stakeholders; 
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• The needs of managers and ‘doers’ in respect of knowledge management - and the 
depth of insight required in respect of the status quo and progress against activities, 
outputs and outcomes, for performance gaps to be addressed; 

• The elements addressed via the ‘CREAM’ criteria (Kusek & Rist, 2004, p.68), which 
specifies that indicators must be: 

o “Clear Precise and unambiguous 

o Relevant Appropriate to the subject at hand 

o Economic Available at a reasonable cost 

o Adequate Provide a sufficient basis to assess performance 

o Monitorable Amenable to independent validation” 

• Time, quality and quantity parameters, where this is possible (AusAID, 2005) - e.g. 
“percent change in revenue collected by 2015”. Through using high levels of 
specificity, the City will be better prepared for the task ahead. 

In identifying indicators, we should be cognisant of those indicators that are currently in 
place within the City, given the cost and resource implications associated with the 
establishment of new indicators, and the collection of aligned data. However, while access to 
information is a valid concern, indicators should not be identified on this basis alone. The 
most critical consideration is whether or not indicators are ‘relevant’ when viewed in the 
context of the planned impact, outcome, output, activity or input. 

It is also important to note that the identification of activities and inputs is more likely to take 
place within a business plan or project chart, instead of falling within the domain of an annual 
or a five-year IDP. As such, many of the templates included here do not address this level of 
detail. 

The GDS provides a set of indicators that serve as valuable starting points for the 
identification of indicators, particularly when planning for M&E in relation to long-term 
impacts and outcomes. These GDS indicators should be reviewed in the context of the 
criteria identified above, and confirmed if relevant. Following this, indicators can then be 
cascaded, with identification of suitable indicators for each level of delivery within the 
outcomes approach. This process allows for confirmation of the relevance of indicators, 
while also aiding self-reflection on operational plans themselves - and areas in which 
adjustments may be needed. An example of the type of indicators provided within the GDS 
are those reflected for the first outcome of “Improved quality of life and development-driven 
resilience for all”. Indicators include, amongst others: the Human Development Index, the 
Poverty Rate and the Food Security Index. These may be broken down further into individual 
index elements, when monitoring delivery at the level of outputs and activities. 

Once indicators have been set, baselines and targets for the relevant timeframe must be 
identified for each indicator, alongside identification of suitable sources of information 
against which changes will be tracked. 
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5.2.2. ‘Where are we at the moment?’ Setting baselines (Step 2) 

Baselines provide a view of “...the situation before you do anything” (Shapiro, 2002, p.28). 
Targets for future levels of achievement cannot be set without a clear sense of the status 
quo. The City’s Group Performance Management Guidelines document notes that “baseline 
information (quantitative or qualitative)... provides data at the beginning of, or just prior to, 
the monitoring and evaluation period. The baseline is used to: 

• Learn about recent levels and patterns of performance on the indicator; and to 

• Gauge subsequent programme, or project performance i.e. future performance 
targets.” (2009, p.33) 

Baselines must be identified for each indicator across all levels reflected in the outcomes 
approach (i.e. impacts, outcomes, outputs, activities and inputs). In identifying baselines, 
focus should be placed on a measure of the status quo, in the context of the defined 
indicator. In the event that the indicator is new, baseline estimates may be formulated 
through: 

• Reviewing statistical reports that present sector-wide standards or levels of delivery, 
in the context of other local government environments (e.g. Stats SA reports); 

• Conducting a brief survey or data collection process, to establish a sound baseline; 

• Identifying a comparable group, for which a baseline does exist. 

Accurate evaluation is difficult when baselines are not in place for each of the identified 
indicators at the initial start of the delivery period (DPME, 2011). While some circumstances 
provide M&E practitioners with no alternative but to commit to the establishment of baselines 
only after the first year within which a new indicator is used, this approach is suboptimal. 
Operating without a baseline jeopardises real analysis of changes in outcomes, as 
movement is made towards the defined targets. It also places the City at risk of setting 
inappropriate targets. Those setting indicators should be conscious of the fact that each 
indicator will need a baseline, and will need to be supported by systems for data collection, 
analysis, reporting and communication. Should indicators change, the baseline will need to 
change accordingly. This may lead to challenges in tracking progress, backlogs and effective 
remedies. Given this fact, it is not advisable for indicators to be changed on a regular basis. 

The table below provides a platform for decision-making on, and the management of 
baseline data in respect of each indicator. Once the approach to baseline data has been 
determined, this framework can also be used for subsequent collection of further information 
relating to each indicator - thereby supporting the tracking of delivery against the defined 
plan. This table is also included in Annexure 5, as a supporting template to the M&E 
Framework. 
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Table 5.1: Integrating indicators with data management (Kusek & Rist, 2004, p.82) - 
(Annexure 5) 


Indicator 

Data 

source 

Data 

collection 

method 

Who will 

collect 
the data? 

Frequency 

of 

collection 

Cost and 
difficulty 
to collect 

Who will 
analyse 
the data? 

Who will 

report 

data? 

Who will 

use the 

data? 

1 









2 









3 










The columns relating to data source and data collection method are as applicable for M&E 
activities, as they are for the initial baseline definition. The section below addresses the 
various approaches to data collection that may be included within an M&E plan. 


5.2.3. What do we want to achieve for each time period? Identifying targets (Step 3) 

Once a baseline has been set for each indicator, a target can be identified for the planning 
period ahead, where this serves as “...a specified objective that indicates the number, timing 
and location of that which is to be realised” (IFAD, 2002, cited in Kusek & Rist, 2004, p.90). 
Targets should be set for each indicator level applicable within the context of the City’s 
planning timeframes (i.e. for indicators addressing the level of ‘impacts’; ‘outcomes’; 
‘outputs’; ‘activities’; and ‘inputs’). As a result, targets will therefore need to be set for 
indicators framed in the context of the City’s long-term GDS, the five-year IDP and Cluster 
plans, the aligned Mayoral Flagship Programmes and the one-year IDP, SDBIP, business 
plans and individual scorecards. In this way, responsibilities for delivery against the various 
targets are cascaded. 

Kusek and Rist (2004) argue that the following should considered when setting targets: 

• The baseline and the level of improvement envisaged - with the target then serving 
as an expression of the baseline state with the improvement included; 

• Past performance in respect of the indicator, or alternatively, related levels of 
delivery; 

• The volume and quality of resources available to support delivery in the period ahead 
- with due consideration of other priorities that may demand resources; 

• Commitments made to stakeholders - including the electorate; 

• The extent to which potential targets are ‘realistic’ - i.e. the extent to which they are 
feasible, when considered in the context of milestones to be reached before the 
target is realised. 

As with individual target setting (CoJ, 2009a), targets at this level should be SMART - : 

S - Simple, clear and understandable 

M - Measurable, in terms of quantity and where possible, quality, money and time 
A - Achievable and agreed 

R - Realistic - within the control of the responsible parties, but challenging 
T - Timely - to reflect current priorities; assessable within the defined reporting period 
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Targets may be stated in the context of: time (e.g. in relation to milestones), quality (with 
quality parameters including, for example, a focus on deliverables that are ‘relevant’, 
‘practical’, ‘legislatively compliant’, etc.), quantity (e.g. total cost, count, an average, etc.) or 
any combination thereof. Quantitative targets may also include “target ranges as opposed 
to single numeric values” (CoJ, 2009a, p.34). 

How firm should targets be? Adhering steadfastly to poorly devised or unrealistic targets is 
counter-productive. During delivery and during the processes of M&E, stakeholders may 
realise that targets, or the indicators against which they are set, are poorly suited to the task. 
In such circumstances, the target and/ or indicator, if necessary, should be revisited, to 
ensure relevance. When targets are defined in the context of a multi-year model, the starting 
point should be the long-term project or deliverable. Each year, targets need to be set and 
potentially revised with this end objective in mind, given learnings from the period that has 
passed. Focus should remain on the long-term target. Once targets have been defined, the 
next step in the M&E planning process requires focus. Step 4 addresses the ‘how’ aspect of 
M&E in the context of targets, baselines and indicators. 


5.2.4. How will we measure and analyse delivery against the defined targets? (Step 4) 

While information collection for M&E purposes only takes place after the completion of the 
planning period, the planning stage must include the identification of appropriate information 
sources or ‘Means of Verification’ (MoV) - i.e. mechanisms through which progress against 
defined targets is to be assessed, for both evaluation and monitoring. Kusek and Rist (2004, 
p.82) provide the following questions, to be considered when identifying an information 
source for baseline data, and for subsequent measurements of progress against indicators, 
following the initiation of delivery: 

• “What are the sources of data? 

• What are the data collection methods? 

• Who will collect the data? 

• How often will the data be collected? 

• What is the cost and difficulty associated with data collection? 

• Who will analyse the data?” 

• Who will use the data?” 

Most M&E activities come with a cost attached. As a result, it is also valuable to consider 
other existing sources of information from which data may be accessed (e.g. research 
bodies or other arms of government). A further consideration relates to the manner in which 
reporting and communication of data will take place - with this leading to additional 
questions: ‘Who will report the data?’ ‘In what format should data be recorded?’ Maintaining 
an awareness of what is needed in terms of an appropriate MoV may aid in fine-tuning MoV 
related decisions. Focusing on practical considerations, such as cost and resource 
requirements and the amount of data that is actually required (i.e. avoiding the collection of 
unnecessary data), will also aid the planning process. 

For information sources to be used appropriately, MoVs will be established at the start of the 
M&E cycle. For both monitoring and evaluation-related activities, consideration may be given 
to formal or less formal methods of data collection, although some aspects of both M&E are 
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legislated and fall within the City’s standard performance management cycles (e.g. in terms 
of monitoring: coaching and daily record keeping of delivery, as may be needed to support 
operations; in terms of evaluation: formal quarterly reviews and annual review). The figure 
below depict various data collection methodology, reflected on a continuum from less- 
structured to more structured in nature. 


Figure 5.5: Varying approaches to data coiiection (cited in Kusek & Rist, 2004, p.85) 
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More input on MoV approaches in the context of monitoring or evaluation is provided below. 


5.2.5. Integrating planning activities into City-wide and Cluster M&E Plans (Step 5) 

Following the completion of Steps 1 to 4, the M&E planning process comes to a conclusion 
through the integration of all elements (indicators, baselines, targets and data collection/ 
analysis methods for M&E - i.e. MoVs) into various levels of integrated M&E plans. These 
may include, for example, a City-wide M&E Plan, a cluster-specific plan, or an inter-cluster 
plan. 

While addressing each of these steps may appear arduous, this concern is balanced by the 
potential for improved delivery that may result from sound M&E planning, and the 
formalisation of this through an integrated plan (with the latter bringing together the results of 
both operational planning and that related to M&E). The figure below provides an overview 
of the M&E planning elements addressed within the planning phase, in the context of 
broader organisational, cluster, departmental/ entity or individual planning process. 
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Figure 5.6: Key steps addressed within the pianning process 
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National and provincial government is obligated to establish three-year and one-year 
evaluation plans (DPME, 2011). The City chooses to adopt a similar approach, to maintain 
appropriate focus. To ensure all the steps identified in the planning process are put into 
place, M&E plans will be developed at a City-wide and cluster level. Proposed templates for 
City-wide and cluster-level M&E planning are included below, and in Annexures 6, 8, 9 and 
10 (for ease of reference). 

In reviewing the templates, please note that the first two templates reflect M&E plans in the 
context of operational plans. Included within these templates is a column relating to 
assumptions, allowing for the capturing of any “assumptions made about conditions which 
could affect the progress or success of the activity, but over which activity managers may 
have no direct control, e.g. price changes, rainfall... [It] is a positive statement of a condition 
that must be met in order for objectives to be achieved... [in contrast with] a negative 
statement of what might prevent objectives being achieved” (AusAID, 2005, p.35). This is 
important, as it allows the theory of change that has been applied as part of the operational 
planning process to be checked and refined, as implementation and the M&E associated 
activities of data collection, analysis and reflection continue. This column also allows for risks 
that may threaten delivery to be identified (regardless of whether these relate to inputs, 
activities, outputs or outcomes), alongside mitigating actions that are to be implemented. 
This will serve to support delivery, through enforcing a reflection on the extent to which the 
envisaged plans are realistic. 

More specifically, the first template included below in Table 5.2 relates to a City-wide M&E 
Pian. The following is noted: 

• The template is inclusive of outcomes and outputs, and a breakdown between 
indicator type, baselines and targets. It does not include activities and inputs, as 
other supporting frameworks will be used to drill down to this level. The proposed 
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M&E Plan template provides for planning within a five-year cycle, with the ideal 
approach to the planning process involving the backwards cascading of impacts 
envisaged within the GDS for 2040, within five-year planning cycles (e.g. 2016-2020; 
2021-2025; 2026-2030; 2031-2035; 2036-2040). This planning is to be revisited and 
refined at each planning period, to ensure relevance - and delivery on the long-term 
objectives. 

• The M&E Plan should also include consideration of the three categories defined in 
respect of outputs - i.e. ‘continuous improvements’; ‘impactful improvements’; ‘broad 
initiatives’ - with a balance across these different categories to be included. The 
table below provides a useful basis for identifying these, in relation to each outcome 
- for follow-through by the cluster. 

• As noted, this template is again reflected in the Annexures (Annexure 6), for ease of 
reference. An additional example of this template has been provided in Annexure 7, 
with preliminary inputs included - identified in the context of outcomes defined in the 
City’s 2040 GDS. 

The second template (Table 5.3) depicts a medium-term (five-year) ciuster-specific M&E 
Pian. This is also included in Annexure 8, attached hereto. The following is noted: 

• In designing the cluster-specific M&E Plan, focus should be placed on the breakdown 
of outcomes, sub-outcomes (where applicable) and outputs in relation the City’s 
clusters (i.e. allowing for an indication of ownership and responsibility for delivery, at 
the level of the City’s four defined clusters). Once again, this should take into 
consideration the three categories of output: ‘continuous improvements’, ‘impactful 
improvements’, and ‘broad initiatives’). 

• This table is translated into an annual cluster-specific M&E Plan - which may be 
implemented at the level of individual clusters, allowing for the identification of 
delivery agents (e.g. departments, entities or other components within the City’s 
structure). This is reflected in Table 5.4 below - and is included in Annexure 9. 

The final M&E planning template is included in Table 5.5 below, and in Annexure 10, for 
ease of reference. This template caters for inter-cluster M&E planning. This is seen as 
important as it is very often in the interface points between various sections of an 
organisation that plans fail to materialise, or are not fully conceptualised. This template 
encourages focus on the interface points between the various clusters (reflected in outputs - 
or outputs and activities, if the latter are deemed necessary at this level of planning). This 
will: 

• Ensure continued awareness of the importance of all within the City operating as a 
team, in support of our long-term vision; and 

• Assist in ensuring identification of those areas where delivery by one party is 
dependent on another. 


The City of Johannesburg’s Monitoring and Evaluation Framework 

48 I Page 


14 May 2012 



Table 5.2: Proposed M&E Plan for City-wide M&E planning (Annexure 6) 


Impact: [Description of 
impact/ goal at a City level] 

City-wide metric 

City-wide annual targets 

Assumptions, risks and 

mitigation 

impiementing Ciuster 

City-wide 

Budget 

Planning cycle: E.g. 
2020-2025 

Key outputs 

32 

o 

'co 

Means of 
Verification 

5-year IDP 

Target 

Baseline 

Key outcomes 

Yr 

-3 

Yr 

-2 

Yr 

-1 

Current 

year 

target 

Yr 

1 

Yr 

2 

Yr 

3 

Yr 

4 

Yr 

5 

Yr 

1 

Yr 

2 

Yr 

3 




















































































































Table 5.3: Proposed M&E Plan for cluster-specific M&E planning (Annexure 8) 


impact: [Description of 
impact/ goal at a City level] 

Ciuster-specific metric 

Ciuster-specific annuai targets 

Assumptions, risks 
and mitigation 

impiementing 

department/ entity/ 

section 

impiementing 

department/ 

entity/ 

section 

budget 

Planning cycle: E.g. 
2020-2025 

Key outputs 

indicator/s 

Means of 
Verification 

5-year iDP 

Target 

Baseiine 

Key outcomes 

Yr 

-3 

Yr 

-2 

Yr 

-1 

Current 

year 

target 

Yr 

1 

Yr 

2 

Yr 

3 

Yr 

4 

Yr 

5 

Yr 

1 

Yr 

2 

Yr 

3 




















































































































^ This could also be structured to include target information - i.e. indicator and exact target. 
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Table 5.4: Proposed M&E Plan for annual M&E planning - at the cluster/ departmental/ 
entity level (Annexure 9) 


Annual plan for: [Year] 

Cluster-specific metric 

Cluster-specific annual 
milestones 

Implementing 
department/ 
entity/ section 

Budget 

Key 

outcomes 

Key outputs 

Activities 

Indicator/s'' 

Means of 

Verification 

Baseline 

Annual 

Target Yr1 

Q1 

Q2 

Q3 

Q4 

Estimate 

Variance 


























































Table 5.5: Proposed M&E Plan for Inter-cluster M&E planning (Annexure 10) 


Impact: [Description of impact/ goal] 

Planning cycle: E.g. 2020 - 2025 


Cluster 1 

Cluster 2 

Cluster 3 

Cluster 4 

Cluster 1 


Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 

2’s areas of focus 

Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 

3’s areas of focus 

Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 
4’s areas of focus 

Cluster 2 

Include the 
contribution to be 
made - at the level of 
output or activity, in 
relation to Cluster Ts 
areas of focus 


Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 

3’s areas of focus 

Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 
4’s areas of focus 

Cluster 3 

Include the 
contribution to be 
made - at the level of 
output or activity, in 
relation to Cluster Ts 
areas of focus 

Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 

2’s areas of focus 


Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 
4’s areas of focus 

Cluster 4 

Include the 
contribution to be 
made - at the level of 
output or activity, in 
relation to Cluster Ts 
areas of focus 

Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 

2’s areas of focus 

Include the 
contribution to be 
made - at the level 
of output or activity, 
in relation to Cluster 

3’s areas of focus 



Together, the above tables provide a mechanism through which to carry out iterative testing 
of each of the details within the M&E hierarchy, and the interface of these with the City’s 
structure and role-players. Once the various M&E Plans are finalised, all groundwork 


This could also be structured to include target information - i.e. indicator and exact target. 
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elements are in place for the City to shift to delivery, with M&E planning then put into 
practice through ongoing monitoring, and time-specific evaluation. These elements are 
addressed below. 


5.3. Phase 2: Conducting monitoring 

Monitoring takes places as an ongoing process, across the cycle of project, departmental, 
entity, cluster or City-wide delivery. The outcomes approach enables the identification of a 
hierarchy of indicators and targets - defined in the context of impacts, outcomes, outputs, 
activities and inputs. Monitoring of delivery is required in respect of each of these levels, with 
identification of the most appropriate monitoring tools and systems being heavily dependent 
on the nature of the indicators confirmed during the planning stage. The foundation for the 
monitoring phase is therefore largely in place when the planning process has been 
concluded. 

The primary focus here is on gathering, collating, inspecting and analysing information, in 
the context of indicators and short, medium and long-term targets. Monitoring activities are 
mostly carried out by internal role-players within the City - with information emerging often 
used as a basis to inform managers and employees on how actions should be redirected to 
support the achievement of defined outputs and outcomes. 

The following key steps are included here in respect of monitoring, and are addressed in the 
sectors that follow: 

• Step 1 : Confirmation of monitoring tools and systems 

• Step 2: Gathering and collation of information 

• Step 3: Analysis of information (please note that this step applies to both M&E - and 
is detailed in the section on evaluation, below) 

5.3.1. Confirmation of monitoring toois and systems (Step 1) 

Section 5.2.4 addresses some of the key questions requiring attention when determining the 
most appropriate tools through which monitoring-related (or evaluation-related) information 
will be gathered and analysed. The choice of tool needs to be directly related to the level of 
the outcomes approach being assessed, alongside consideration of other factors, such as 
the audience that will be receiving feedback, and the level of detail required. 

A more elaborate ‘how to’ guide in respect of decision-making relating to monitoring tools 
and methodology will be included the M&E Handbook. It is useful here to reflect, however, 
on the fact that the most monitoring methodology will often include a combination or a 
balance of different types of tools, across three categories: monitoring tools focused on 
reporting and analysis; monitoring tools focused on validation (i.e. verification of findings and 
results); and monitoring tools that emphasise participation. The table below provides an 
overview of possible tools - clustered within these categories. 
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Table 5.6: Establishing the right mix of monitoring tools (UNDP, n.d.(b), p.36) 


Reporting and analysis 

Validation 

Participation 

• Annual project report (APR) 

• Progress and/or quarterly reports 

• Work plans 

• Project/programme delivery reports 
and combined delivery reports 

• Substantive project documentation 

• Field visits 

• Spot-check visits 

• External assessments/ 
monitoring 

• Client surveys 

• Evaluations 

• Outcome groups 

• Steering committees/ 
mechanisms 

• Stakeholder meetings 

• Focus group meetings 

• Annual review 


In applying the above, it is useful to note that monitoring the achievement of outcomes, as 
opposed to simply monitoring delivery on activities or inputs, may, for example, require a 
balanced combination of these. The ultimate audience of ‘outcomes’ is often the recipient - 
necessitating some of the more participatory forms of monitoring, alongside mechanisms for 
validation. In contrast, when monitoring inputs, focus may instead be placed on reports and 
work plans - supported by some form of verification (e.g. a spot check). 

Monitoring activities will also need to be supported by a monitoring system, which will, in the 
case of the City, include an IT database through which indicators, baseline information, data 
and analysis can be stored, maintained and readily accessed. Kusek and Rist (2004, p.104) 
suggest that the following questions need to be addressed in setting up this system: 

• What data will be collected? (i.e. source) 

• How often will data be collected? (i.e. frequency) 

• How will data be collected? (i.e. methodology) 

• Who will collect the data? 

• Who will report on the data? 

• For whom is data collected? 

In accordance with good practice, the data collection and information management system 
(i.e. the monitoring system) within the City will need to align with a further set of principles, 
outlined in the box below. 

Box 5.1: Principles fora sound monitoring system (Adapted from Kusek & Rist, 2004) 


• Ownership: This requires that every person, across all levels and areas within the City, 
feels a sense of ownership and an understanding of the benefit of data emerging - with 
ownership and input by all aiding data quality, management and usefulness. 

• Management: All hold a clear view of who will manage information, where, and how. 
Poor data management may lead to duplications in effort, wastage of scarce resources 
(including funding), data being received too late for decision-making purposes - or not 
being used at all (e.g. when there is limited knowledge of data being collected, or where 
it is located). Data management may cover a wide range of factors - e.g. hard and soft 
copy storage of content; necessary controls; and supporting technology (e.g. the IT 
platform). 
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• Maintenance: Any information system and the data within it will need to be maintained. 
Focus needs to be placed on ensuring ongoing collection of quality data, and the 
appropriate flow of information across the City, as needed. 

• Credibility: System credibility is critical, if results and findings are to be used, and efforts 
and resources therefore optimised. To build and maintain credibility, data needs to 
adhere to the principles of reliability (with indicators measured in the same way, over 
time), validity (with indicators measuring, as closely as possible, the performance or 
aspect they propose to measure) and timeliness (with consideration of frequency of 
data collection, how recently it has been collected, and the relevance thereof to 
decisions). Findings and analysis need to be reported in all cases - with both positive 
and negative findings reported transparently. 


The above will require dedicated resources (e.g. financial, people, IT support, research 
specialists, communication mechanisms, etc.), to ensure sustainability. Given that delivery 
by the City’s long, medium and short-term objectives ultimately depends on this, the 
importance of establishing appropriate systems and processes for data management and 
maintenance cannot be over-emphasised. While the SDBIP provides some support in this 
respect, alongside the more manual forms of data collection undertaken as part of the City’s 
current performance management processes, we recognise the need for further work in this 
regard. In particular, the integration of credible data sources across the City (at best) 
requires focus - with an interim measure including the establishment of more standardised 
processes to facilitate timely access to accurate data, for stakeholders across the City. 


5.3.2. Gathering and collating data in the context of monitoring (Step 2) 

While both M&E activities involve data gathering and analysis, the purpose of data 
gathering, methods and responsibilities may vary between these activities (and between the 
different stages of the cycle at which they may occur). In particular, data gathered during 
monitoring activities may well focus on addressing very different questions to those attended 
to by an evaluation exercise. 

As with an evaluation exercise, in the context of monitoring, the nature of data and 
information to be gathered will already be defined during the planning stage. This step is 
therefore about putting these decisions into practice. Data may come in many forms - e.g. 
CAPEX or OPEX data, statistics from service delivery activities, performance audits, 
financial audits, project completion reports, analysis carried out as part of the City’s 
knowledge management initiatives, etc. Understanding the sources of information that 
already exist is important - even if access needs to take place via other platforms and 
providers such as non-governmental organisations (NGOs), research bodies and technical 
specialists. 

In gathering data, focus is to be placed on data that is relevant, accessible, timely, 
understandable and accurate. Understanding how the data will be used impacts directly on 
the nature of information collected. For example, information gathered as part of a 
monitoring process that is focused on project improvement in the case of a fairly minor 
project may represent a smaller data set than that which would be collected to aid decision- 
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making in respect of a significant financial investment. In line with the outcomes-based 
hierarchy, data may be gathered at the level of implementation (inputs, activities and 
outputs), or at the level of results (outcomes and impacts). The latter is addressed more fully 
below in relation to evaluation, given that this is usually the level at which evaluation 
activities are targeted. 

Errors in data collection may relate to sampling errors (where the inappropriate or 
incomplete sample has been used when collecting data), or non-sampling errors (i.e. 
information excluded). The latter may emerge due to: 

• Interviewer bias (e.g. an interviewer who does not remain objective, is not motivated 
or does not have adequate skills); 

• Inadequate methods (e.g. inappropriate or incomplete questionnaires; poor data 
collection tools); 

• Processing errors (e.g. data incompletely or erroneously captured, and inadequate 
quality assurance); or 

• Non-response bias (e.g. where questions are not easily understood). 

Mechanisms through which these may be avoided are detailed below. 


Table 5.7: Data collection errors and ways to avoid them (IFAD, 2002b, p.19) 


Common errors 

Ways to avoid them 

Interviewer bias 

• Make sure everyone understands the purpose of each method. 

• Make sure everyone knows exactiy what data she/he is coiiecting - ciarify 
units, whom to speak with or where to go for data, and the frequency of 
coiiection. 

• Practise interviewing and faciiitation techniques. 

• Brainstorm about possibie probiems that might occur and agree on various 
ways to avoid them or deai with them shouid they occur. 

Processing errors 
caused by poor 
documentation of 
data 

• Standardise formats for documentation. 

• Practise formats with the users and adapt the formats if necessary. 

• Computerise as soon as possibie after data coiiection and check the data 
entries. 

• Have enough materiai to record aii responses and avoid iosing data. 

Non-response 

bias 

• Pre-test questions and methods. 

• Present methods and questions (and especiaiiy their purpose) cieariy and 
confirm that peopie have understood. 

• Use iocai terms. 


5.4. Phase 3: Conducting evaluations 

As noted previously, evaluations are carried out with various objectives in mind, as reflected 
by the stage during which evaluations are conducted within the lifecycle of a project, 
programme or plan. It is argued that evaluations are “characterised by events (e.g. surveys, 
studies, missions) rather than day-to-day data collection. Results-oriented evaluation 
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focuses on outcomes and impacts and adds to and builds on monitoring information. It 
assesses overall performance, focusing on positive or negative changes in beneficiary 
behaviour or status occurring as a result of an operation” (WFP, n.d., p.9). As with 
monitoring, evaluation can be conceptualised within the context of the following steps: 

• Step 1 : Confirmation of evaluation tools and systems 

• Step 2: Gathering and collation of information 

• Step 3: Analysis of information 

5.4.1. Confirmation of evaiuation toois and systems (Step 1) 

In determining the most appropriate tools or methodologies to use when carrying out 
evaluation exercises, the key consideration relates to the purpose of the evaluation itself. As 
noted in Chapter 2, an evaluation may focus on questions relating to efficiency, 
effectiveness, relevance, impact or sustainability. The focus of an evaluation exercise is also 
dependent on whether the evaluation is formative, summative, or conducted during the life- 
cycle of the project, programme or policy. In the City’s context, many evaluations occur in 
the context of particular evaluation periods, as detailed in Figure 3.9. In particular, 
evaluations may take place within the context of: 

• Bi-annual evaluations conducted in respect of delivery on departmental scorecards; 

• Quarterly and annual reviews conducted in respect of annual business plans; 

• Quarterly reviews and annual evaluation conducted in the context of the annual 
SDBIP; 

• Bi-annual reviews conducted in the context of the annual IDP; 

• Mid-term reviews undertaken in respect of the five year cluster plans and the long- 
term GDS; 

• Impacts and outcomes defined in relation to the above - measured through regular 
evaluations such as the City’s Customer Satisfaction Survey. 

The DPME provides a further refinement of the possible types of evaluation, with the 
intention of establishing a common language and a shared set of standard procedures. Six 
types of evaluation are identified - explained through the figure and table below. 
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Figure 5.7: Varying approaches to data coiiections (DPME, 2011, p.8) 


Economic Evaluati 
What are the 
cost-benefits? 



IHPUTS 


DESIGN 


Diagnostic " 

What is the underlying 
situation and root 
causes of the problem? 




Impact evaluation 
Has the intervention had 
impact at outcome and 
impact level, and why? 


Implementation 
evaluation 
What is happening 
and why ? 


r Design evaluation 
^ Assess the theory of 
change 


These six types of evaluation are explained further in the table below - with examples of 
verification methods included. 


Tabie 5.8: Summary of types of evaluation across government (Adapted from: DPME, 

2011, pp.1 0-11) 


Type 
of eval 

Covers 

Timing 

Exampies of 
evaiuation 
methodoiogy® 

Diagnostic Evaiuation 

This is preparatory research (often called ex- 
ante evaluation) to ascertain the current 
situation prior to an intervention and to inform 
intervention design. It identifies what is already 
known about the issues at hand, the problems 
and opportunities to be addressed, causes and 
consequence, including those that the 
intervention is unlikely to deliver, and so the 
likely effectiveness of different policy options. 
This enables you to draw up the theory of 
change before you design the intervention. 

At key stages 
prior to design 
or re-planning 

• Formal surveys 

• Stakeholder 
analysis 

• Secondary data - 
e.g. statistical 
analyses; 
interviews; focus 
groups; literature 
reviews 


^ Please note: examples here are drawn from the World Bank (2004) - with varied approaches dependent on 
purpose of evaluation and circumstances within which it is conducted. 
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Type 
of eval 



Examples of 

Covers 

Timing 

evaluation 



methodology® 


Used to analyse the theory of change, inner 

After an 

• Quantitative 


logic and consistency of the programme, 

intervention has 

statistics (e.g. 


either before a programme starts, or during 

been designed. 

community survey; 

C 

implementation to see whether the theory of 

in first year, and 

household survey) 

'i? 

change appears to be working. This is quick to 

possibly later 

• Qualitative methods 

D 

do and uses only secondary information and 


such as semi- 

CO 

> 

should be used for all new programmes. It 


structured and 

<1> 

should check that the outcomes chain 


structured 

O) 

culminates in impacts that address the main 


interviews. 

(0 

<1> 

situation that gave rise to the intervention. 


observation 

Q 

even if the intervention won’t be held fully 
accountable for these ultimate outcomes. It 
also assesses the quality of the indicators and 
the assumptions. 


records, field notes, 
and focus groups 
transcripts 


Aims to evaluate whether an intervention’s 

Once or several 

• Secondary data - 

_o 

operational mechanisms support 

times during the 

e.g. statistical 

c3 

achievement or not and understand why. 

intervention 

analyses; 

CO 

Looks at activities, outputs, and outcomes, use 


interviews; focus 

> 

<1> 

of resources and the causal links. It builds on 


groups discussions; 

c 

Q 

existing monitoring systems, and is applied 


direct observation; 

'i? 

during programme operation to improve the 


literature reviews 

c 

efficiency and efficacy of operational 


• Field work - e.g. 

<1> 

p 

processes. It also assesses the quality of the 


participant 

0) 

indicators and assumptions. This can be rapid 


observation; data 

Q. 

c 

primarily using secondary data or in-depth with 


collection, and 


extensive field work. 


survey research 


Seeks to measure changes in outcomes and 

Designed early 

• Quasi-experimental 

c 

the well-being of the target population that are 

on, baseline 

design with before 

.2 

attributable to a specific intervention. Its 

implemented 

and after 

CO 

3 

purpose is to inform high-level officials on the 

early, impact 

comparisons of 

CO 

extent to which an intervention should be 

checked at key 

project and control 

0) 

continued or not, and if there are any potential 

stages e.g. 3/5 

populations 

o 

CO 

modifications needed. This kind of evaluation 

years 

• Ex-post comparison 

Q. 

c 

is implemented on a case-by-case basis. 


of project and non- 




equivalent control 
group 


Economic evaluation considers whether the 

At any stage 

• CEA 


costs of a policy or programme have been 
outweighed by the benefits. Types of economic 


• CBA 


evaluation include: 



.2 

• cost-effectiveness analysis (CEA), which 



CO 

values the costs of implementing and 



CO 

delivering the policy, and relates this amount 



> 

0) 

to the total quantity of outcome generated, to 



u 

produce a “cost per unit of outcome” 



E 

estimate (e.g. cost per additional individual 



c 

o 

placed in employment); and 



u 

UJ 

• cost-benefit analysis (CBA), which goes 




further than CEA in placing a monetary value 
on the changes in outcomes as well (e.g. the 
value of placing an additional individual in 
employment). 
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Type 
of eval 

Covers 

Timing 

Examples of 
evaluation 
methodology® 

Evaluation 

synthesis 

Synthesising the resuits of a range of 
evaiuations to generaiise findings across 
government e.g. a function such as suppiy 
chain, a sector, or a cross-cutting issue such 
as capacity. DPME wiii undertake evaiuation 
synthesis based on the evaiuations in the 
nationai evaiuation pian and do an annuai 
report on evaiuation. 

After a number 
of evaiuations 
are compieted 

• Annuai report on 
evaiuation findings 
across the City - 
synthesising aii 
evaiuations 


It is through careful consideration of the above, and aligned issues such as budget, risk and 
impact of decisions emerging from an evaluation, that the final decision relating to the 
evaluation will be taken. Given that evaluations often tend to be more formal than monitoring 
processes, their rollout processes are frequently more plan-based. The approach to 
evaluation-related data gathering is addressed below. 


5.4.2. Gathering and collating data in the context of evaluation (Step 2) 

Many of the principles applicable to data collection for monitoring purposes apply in the case 
of evaluation. However, given that evaluations are often undertaken by a service provider 
external to the project or process (even though this may be a service provider internal to the 
City), it may be appropriate to develop a Terms of Reference (ToR) for data collection and 
analysis. Shapiro (2002) notes that this should typically address the following: 

• The background to the evaluation - e.g. the context of the project or the City’s 
context, the problem or issue identified, actions taken, processes followed and the 
rationale for carrying out an evaluation (addressing the question of whether it is 
focused on issues of relevance, efficiency, effectiveness, utility or sustainability) 

• The main objective of the evaluation - i.e. what the evaluation should achieve 

• Outputs - e.g. reports to be produced, feedback material, etc. 

• Major issues Key questions (themes or elements) that the evaluation must focus on 
answering (e.g.: achievements of a project at a site) 

• Approach- Methodological considerations, including a focus on the key principles 
and parameters in respect of the research methodology, and techniques to be used 

• Organisation/ logistics and management considerations - e.g. composition of 
the evaluation team, and resource considerations 

• Preliminary work plan - including a time schedule 

In assessing the appropriateness or quality of a proposed or completed evaluation, a 
number of criteria or principles should be considered - as addressed in the box below. 
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Box 5.2: Assessment of an evaluation exercise: key criteria (Adapted from Kusek & 
Rist, 2004) 


• Impartiality: The evaluation should be free from bias or interference. 

• Usefulness: Information needs to be timely, relevant and documented in a manner that 
is user-friendly. 

• Technical adequacy: Information needs to, amongst other things, “meet relevant 
technical standards - appropriate design, correct sampling procedures, accurate wording 
of questionnaires and interview guides, appropriate statistical or content analysis, and 
adequate support for conclusions and recommendations...” (p.127). 

• Stakeholder involvement: For stakeholders to view evaluation findings as credible, 
trustworthy and useful for driving action, they should be involved in the evaluation effort 
or process. 

• Feedback: Ensuring timeous, appropriate and user-friendly feedback is critical to the 
credibility of the evaluation itself. Identifying how evaluation information will be 
communicated to the target audience aids appropriate data collection and analysis. 

• Value for money: The cost of the evaluation should be appropriate to the size of the 
intervention itself. Data that will not be used or of value to the City’s stakeholders should 
not be collected. 


In cases of large evaluations, the DMPE recommends the establishment of an advisory 
group and a steering group to ensure: ownership of the evaluation and roll-out according to 
the ToR (i.e. via the steering group); sound technical input, expertise and advice (i.e. via the 
advisory group) 

Analysis and recording of information emerging from M&E activities is addressed below. 


5.4.3. Analysis of information (Step 3 - applicable to both monitoring and evaluation) 

The objective of analysis is to “transform data into credible evidence about the ...intervention 
and its performance. (Ministry of Foreign Affairs of Denmark, 2006, p.72).The task of data 
analysis is one that is applicable in respect of all aspects of M&E. Regular analysis of 
implementation data as part of a monitoring process may assist in improving performance 
during the delivery of outputs and associated activities. It may allow for the identification of 
trends, challenges, risks and areas of success. Data analysed as part of the monitoring 
cycle, however, is often not useful in providing information of why changes are occurring - 
with this being a function of evaluation. Regardless of whether data is to be collected and 
analysed in support of a monitoring or evaluation related activity, the following steps may 
apply: 


• Review the indicators identified for the monitoring or evaluation process; 

• Ensure data is coliected with these indicators in mind (i.e. data is relevant); 

• Establish a structure for the analysis - e.g. in terms of concerns, ideas or themes; 

• Organise the data within the context of this structure, in preparation for analysis; 
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• Focus on patterns, varied forms of interpretation or trends; and 

• Document the findings, and establish conclusions and recommendations 

(Shapiro, 2002). 

The figure below represents the process of analysis in the context of an evaluation: 

Figure 5.8: The analysis process in the context of an evaluation (Ministry of Foreign 
Affairs of Denmark, 2006, p.74) 
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Overall 
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While analysis in the context of a monitoring exercise will take place in respect of the 
predefined indicators included in the M&E Plan, an evaluation exercise may address 
different questions, as depicted in the figure above. The evaluation instruments, analyses 
and findings will ultimately be directly dependent on the nature of questions under review. 

The final phases of the M&E Framework relate to reporting of findings, and 
communication and feedback. These areas are addressed below, alongside input on the 
importance of applying findings and learnings. Through the latter, the M&E planning cycle 
and contents can be revisited within a new timeframe, with improvements incorporated as a 
result of insights gained via M&E undertakings. 


5.5. Phase 4: Reporting on M&E 

The M&E Framework will only be of value if findings are reported on and put into action, 
where necessary. With the M&E Plans and the Indicator Framework depicting stakeholder 
needs and interests, reporting formats, frequency of reporting and other key details, the path 
has already been prepared for reporting on findings, during the first phase of the M&E 
process. 

While unique types and forms of reports may be agreed at the start of a planning cycle, it 
should also be noted that there are a set of pre-defined reporting mechanisms in place within 
the City, many of which are legislated - while others represent good practice that has 
evolved within the City over time. These reporting mechanisms are reflected on in the table 
below, with input included on reporting frequency, reporting responsibility and review 
responsibility included. 
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Table 5.9: City-wide review and reporting matrix (Adapted from CoJ, 2009b, p.37) 


Plan 

Frequency and nature of reviews 
and reporting 

Reporting 

responsibility 

Reviewing responsibility 

GDS 

5-year Assessment Report; 

Mid-term (2.5 year) review 

City Manager 
(CM) 

Legisiature (which 
inciudes the Executive 
Mayor [EM], and is 
supported by Group 

Strategy Poiicy and 
Reiations or GSPR) 

Five-year IDP 

5-year Assessment Report; 

Mid-term (2.5 year) Performance 
Assessment Report; 

Annuai Reports 

CM 

Johannesburg 

Performance Audit 
Committee (JPAC) - 
which inciudes the EM, 
and is supported by GSPR 

Five-year 
Cluster Plans 

5-year Assessment Report; 

Mid-term (2.5 year) Performance 
Assessment Report; 

Annuai Reports 

Ciuster ieads, 
supported by 
ciuster 
members 

Members of the Mayorai 
Committee (MMC), 
through the MMC’s 
quarteriies 

Annual IDP 

Mid-year Report; 

Quarteriy Reports 

CM 

Johannesburg 

Performance Audit 
Committee (JPAC) - 
which inciudes the EM, 
and is supported by Group 
Strategy Poiicy and 
Reiations (GSPR) 

SDBIP 

Quarteriy reports 

CM 

JPAC (which inciudes the 
EM, and is supported by 
GSPR) 

Departmental 

Business 

Plan 

Annuai evaiuation report; 

Quarteriy review report 

Executive 

Directors 

(EDs) 

CM, with inputs from the 
reievant MMC (with GSPR 
support) 

Municipal 

Entity 

Business 

Plan 

Annuai evaiuation report; 

Quarteriy review report 

Managing 

Directors 

(MDs)/ Chief 

Executive 

Qfficers 

(CEQs) 

The Board, with inputs 
from the reievant MMC - 
with support from the 
Sharehoiders Unit (SHU) 


Monitoring reports such as the quarterly review report aid in building an understanding of 
progress and delivery in the context of business plans and the SDBIP, thereby ensuring 
ongoing strategy-aligned implementation. In contrast, evaluation reports such as the Mid- 
Term Performance Assessment Report (undertaken in respect of the five-year IDP and 
cluster plans) represent time-specific analyses of commitments delivered, and those that 
remain unmet. As such, the City’s annual, five-year and longer-term planning cycle 
effectively includes elements of a predefined M&E plan. A more refined M&E system in the 
City will allow for nuanced reflection of types of reporting, audience, purpose, format and 
frequency - thereby aiding more optimal results that are aligned with their intended use. 

In addition to the above reporting mechanisms, the City is also obligated to report on 
progress and delivery, as per the provisions set out in National Treasury’s Municipal Finance 
Management Act (MFMA) Circular No. 1 1 (in respect of the Annual Report) and its 2007 


The City of Johannesburg’s Monitoring and Evaluation Framework 

61 I Page 


14 May 2012 


FMPPI. Reporting mechanisms included here have been designed with these considerations 
factored in. GPG (2012, p. 26) also notes the duty to report on the following: 

• “...National Sector Targets and Indicators 

• ...Service Delivery Improvement 

• ...Batho Pele Targets 

• ...Transversal Issue Targets (Gender, Youth, PWD etc.) 

• ...Employment Equity Targets 

• ...Financial Performance”. 

A City-wide reporting template is included below and reflected in Annexure 11, for 
completeness. A proposed reporting template for Cluster-related delivery follows, which is 
replicated in Annexure 12, for ease of reference. 


Table 5.10: City-wide review and reporting matrix (Adapted from GPG, 2012, p.27) - 
Annexure 11 


Impact: [Description of 
impact/ goal at a City 
level] 

City-wide metric 

Progress Report 

Budget 

Utilisation 

Comments 

Planning cycle: 

Eg. 2020 -2025 

Key outputs 

O 

'CB 

‘•u 

c 

> 

o 

S 

5-year IDP 
Target 

Baseline 

Annual target 

Quarterly 

targets 

Quarterly 
Target 
Achieved 
(Yes/ No) 

If ‘Yes', provide 
evidence 

If ‘No’, state 
Intended 
mitigation 

Estimate 

Actual 

Variance 

Key outcomes 








Q1 







Q2 







Q3 







Q4 














Q1 







Q2 







Q3 







Q4 








Table 5.11: Cluster-related reporting template (Adapted from GPG, 2012, p.28) - 
Annexure 12 


Impact: [Description of impact/ goal 
at a City level] 

City-wide metric 

Progress Report 

Budget 

Utilisation 

Comments 

Planning cycle: 

Eg. 2020 -2025 

Key outputs 

Implementing 

department/ 

entity/ 

section 

Indicator/s' 

> 

O 

S 

5-year IDP 
Target 

Baseline 

Annual target 

Quarterly 

targets 

Quarterly 
Target 
Achieved 
(Yes/ No) 

If ‘Yes’, provide 
evidence 

If ‘No’, state 
intended 
mitigation 

Estimate 

Actual 

Variance 

Key outcomes 









Q1 







Q2 







Q3 







Q4 















Q1 







Q2 







Q3 








® This could also be structured to include target information - i.e. indicator and exact target. 
This could also be structured to include target information - i.e. indicator and exact target. 
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Impact: [Description of impact/ goal 
at a City level] 



Budget 

Utilisation 


City-wide metric 

Progress Report 

Comments 

Planning cycle: 

E.g. 2020 -2025 

Key outcomes 

Key outputs 

Implementing 

department/ 

entity/ 

section 

Indicator/s' 

s 

5-year IDP 
Target 

Baseline 

Annual target 

Quarterly 

targets 

Quarterly 
Target 
Achieved 
(Yes/ No) 

If ‘Yes’, provide 
evidence 

If ‘No’, state 
intended 
mitigation 

Estimate 

Actual 

Variance 









Q4 








5.6. Phase 5: Communication and feedback 

While the reporting templates included above will allow for standardised and consistent 
recording of findings across the City, the communication of M&E findings to the City’s wider 
range of stakeholders - and the provision of any forms of feedback committed to - may 
require the application of a more stakeholder-relevant approach. To ensure maximum value 
is received when communicating M&E findings, it is useful to consider: 

• What the relevant stakeholder’s preferred mode of communication is; 

• How stakeholders will be using the information; 

• What stakeholder expectations may be in respect of reporting; 

• Which format of feedback would best meet stakeholder needs; and 

• How frequently reporting should take place. 

Information presented should be relevant to the audience. For those in the upper levels of 
the City’s structures, focus may be placed on less detailed data, and analysis at the level of 
outcomes and impacts - in contrast with the information received by those responsible for 
operations (Kusek & Rist, 2004). A communication strategy aligned to the M&E Plan will 
assist in ensuring follow-through in this regard. In this way, the credibility of the system and 
those who manage it will be supported. Stakeholders will also be afforded the opportunity to 
gain a real understanding and appreciation of the efforts, achievements and challenges 
faced by the City. Importantly, information emerging will also be accessible in a form that will 
allow for it to be optimally put to use in driving outcomes achievement. 

A variety of communication methods may be considered, to ensure the most effective 
sharing of outcomes - e.g. verbal feedback, written feedback, written summaries, visual 
displays, or audio-visual approaches. Kusek and Rist (2004) provide further input on useful 
ways to communicate analyses and findings, for impact and improved value: 

• Tailor information and presentation formats to suit the audience and their needs, 
(e.g. Council may be more interested in outcomes and outputs, than in activities 
performed); 

• Provide comparisons of data relating to performance over time; 

• Avoid including large “data dumps”, where these are not required; 

• Present data in a clear and understandable format, avoiding jargon and 
inappropriate detail; 

• When presenting findings and recommendations, include input on the implications 
of recommendations, to assist in recommendations being implemented to drive 
improvements; 
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• Where analysis findings are negative, ensure these even negative findings are stiil 
communicated, given that this is essential for the integrity and usefulness of the 
M&E system (e.g. driving improvements), and for credibility of the system and its 
deliverables with stakeholders. Including a focus on explanations and possible 
recommendations would also be important. 

Who within the City should receive feedback on the findings of the M&E Framework? This 
question relates directly to the role each individual within the City plays, in ensuring a 
successful focus on both the M&E process, and more specifically, on delivery against 
targets. Some may argue that feedback is only relevant for and applicable to those within the 
City’s management team? This is addressed in Box 5.4 below. 

Box 5.3: Understanding the role of all players within the City, in driving delivery 


Integrated planning requires a focus on the entire process of delivery - from the simple, 
short-term actions specific to individual performance, to the creation of the end outcome or 
the envisaged ‘whole’, and through it, the desired impact. This alignment is represented 
symbolically through the diagram below - with the precision associated with constructing, 
testing, aligning, servicing and quality assuring a vehicle viewed as equally applicable in the 
context of the actions deemed necessary when establishing a smoothly functioning City. 

For City-wide delivery to succeed in the context of impacts, outcomes, outputs, activities and 
inputs, all within the City need to understand how their individual areas of responsibility and 
the associated deliverables contribute to the achievement of City-wide goals. Maintaining an 
understanding amongst all players within the City of the organisation’s goals, and how each 
small but aligned action contributes to the outputs and outcomes associated with these 
goals, is critical for the achievement of City-wide results. 

It is evident that the end outcome (a smoothly functioning, sound, fully operational City that 
is able to achieve its desired long-term vision) depends on all parts working together, with 
regular refinement, ‘maintenance’, and amendment. When building a car, if even one minor 
part (e.g. a bolt) is missing, the quality of the end outcome (and the ability to function) will be 
affected. 

Maintaining a focus on delivery at every level is therefore critical. Building this understanding 
of delivery in the context of the City may lead to an improved team focus, while also 
deepening the association held between individual roles, scorecards, business and cluster 
plans and the City-wide IDP. If the City’s M&E mechanisms are able to build a greater 
understanding of how all are part of this bigger ‘whole’ through continual feedback on 
delivery in respect of all levels within the outcomes approach, a larger number of 
stakeholders may engage more fully in the collectively-held vision of the future. 
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Figure 5.9: Communicating the impact of individuai contributions, for organisationai 
success 





Like a complicated machine (e.g. a racing car), every organisation is made 
up of many different parts -different sections, systems, processes and 
people - each playing a different role, to achieve a single vision. 

V V 


All parts come 
from different 
places 

V J 


I The parts do not 
I look the same 


4 


When the parts work together, there is movement. Each part contributes in its unique way. 
Each part has a role to play. 

Some parts are clearly visible; others are not. 

All are important. 



< *> 

Losing one part of the puzzle means immobility: 
delays; challenges; possible failure... 

A destination unreached. 

V / 


When operating together, as an 
integrated, cooperative 
organisation, 
we are unstoppable. 


5.7. Applying analysis and findings emerging from the M&E cycle 
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The value of an M&E Framework is only fully realised when analysis and findings are applied 
back to practice, in the context of projects or day-to-day delivery. Findings emerging from 
M&E activities may support performance improvements, and ultimately, the creation of a 
learning organisation - through providing regular insights and reflection. Building a learning 
culture requires all, across the City, to understand and participate actively in action, analysis 
and reflection - while demonstrating ownership of the M&E concept and system. 

In reflecting on the use of M&E findings, it is noted that they may be used: 

• “To demonstrate accountability — delivering on political promises made to citizenry 
and other stakeholders 

• To convince — using evidence from findings 

• To educate — reporting findings to help organizational learning 

• To explore and investigate — seeing what works, what does not, and why 

• To document — recording and creating an institutional memory 

• To involve — engaging stakeholders through a participatory process 

• To gain support — demonstrating results to help gain support among stakeholders 

• To promote understanding — reporting results to enhance understanding of projects, 
programs, and policies.” (Kusek & Rist, 2004, p. 130) 

The application of findings will be encouraged through involving a wider range of 
stakeholders in the M&E and analysis process itself - e.g. via a public participation and 
engagement process such as the GDS Outreach. In addition, strengthening bodies involved 
in oversight within the City and those driving the M&E system itself will also provide the 
platform for improved application of findings. 

The chapter that follows addresses the manner in which M&E issues interface with the 
domain of governance and oversight in the City. The final chapter addresses the 
requirements for successful roll-out of the M&E Framework in full. 
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6. M&E in the context of governance and oversight 


The chapter set out below addresses M&E issues in the context of governance and 
oversight - given the closely interrelated nature of these issues. Input is included here M&E 
in relation to roles and responsibilities within the City, reporting arrangements and other 
related factors. 

6.1 . The role of M&E in supporting good governance and stakeholder engagement 

M&E systems and activities serve to support effective and meaningful stakeholder 
engagement, which in turn is critical for good governance. King III notes that: “In reality, the 
ultimate compliance officer is not the company’s compliance officer or a bureaucrat ensuring 
compliance with statutory provisions, but the stakeholders (2009, p.7). For the City’s 
stakeholders (including residents and citizens, amongst others) to hold the City accountable 
- and for the City to demonstrate transparency in its dealings with stakeholders - accurate 
information is necessary. Meaningful and constructive stakeholder engagement requires 
access to current information and facts. M&E approaches provide an iterative platform for 
information gathering, analysis and distribution (alongside and supported by stakeholder 
engagement) - thereby aiding organisational success and improved stakeholder 
engagement. 

Stakeholder engagement - and more specifically, public participation - forms a cornerstone 
of local government practice. Public participation processes assist the City in identifying and 
focusing on ‘the right things’, through processes of collective engagement. Annual public 
participation processes undertaken by the City provide input into our five-year and one-year 
IDPs, while ongoing ward-level engagements support the democratic machinery, allowing for 
regular two-way communication and the representation of localised needs. All engagements 
are however meaningless if the needs identified and the subsequent commitments made are 
not followed through with rigorous M&E activities, to monitor delivery and evaluate outcomes 
achieved - thereby closing the public participation loop. 

Recent events within the City have also increased awareness of the need for further 
refinements to the City’s M&E systems. While focusing on meaningful M&E is essential if the 
City is to deliver on the long-term commitments outlined within the Joburg 2040 GDS, the 
process leading to the finalisation of the Joburg 2040 GDS was, in itself, of great value. The 
intensive stakeholder engagement process - the GDS Outreach - culminated in the 
development of the final strategy. The City’s citizens and stakeholders responded favourably 
to the engagement process, flagging both their interest in collaborating with the City as 
active citizens, to ensure optimal outcomes for all - and their desire for continued 
communication, feedback and dialogue in respect of delivery issues. The City acknowledged 
the importance of the GDS Outreach process itself, and the opportunity it presented for 
stakeholder interaction with the City, and for different voices to be heard. If the City is to 
ensure that the momentum from the Outreach process is maintained, continued stakeholder 
engagement is necessary - with monitoring, evaluation and the related activity of ongoing 
communication and reporting providing the key ingredients for informed two-way 
engagement. In addition, a solid understanding of the status quo at each point in time, 
represented through a set of defined facts, is necessary for the City to account openly with 
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its stakeholders in terms of successes, achievements, areas of poor performance, 
challenges and deliverables requiring stakeholder involvement. 


6.2. Users and uses of information emerging from the M&E Framework 

In ensuring that the M&E Framework is able to meet the objectives listed above, it is 
important to identify how information gathered through the system will be used, by whom, 
and for what purposes. Clarifying the type and flow of different forms of information, the 
audience, and the manner in which information will be used maximises the potential value of 
such a system. Information emerging from M&E activities may be of relevance as follows: 

• At an operational level, within specific departments, units or entities in the 
City: With information gathered driving delivery, keeping team members on track, 
enabling the identification of risks, supporting improvements in both project delivery 
and management - and facilitating learning. 

• At a City level (specifically, in respect of the Executive): With M&E tools 
supporting the respective heads who are accountable for delivery and for sound 
governance - enabling improved delivery of management, accounting, strategic 
analysis, oversight, decision-making, policy, planning and budgeting functions. 

• At the level of political representatives (specifically, the Legislature): With 
information from M&E activities enabling meaningful debate in the context of 
stakeholder commitments, targeted interaction with the Executive, informed 
engagement with constituencies, information sharing, and fact-based planning, 
decision-making and oversight. 

• At a national and provincial level: With bodies such as the Department of 
Cooperative Governance and Traditional Affairs (CoGTA), National Treasury, 
Statistics South Africa (Stats SA) and the Gauteng Provincial Government (GPG) 
playing a role in gathering and analysing information, and understanding cross- 
sphere delivery and risks. Information emerging from the M&E activities will be of 
value here in supporting the identification of appropriate plans of action and budget 
allocations, from a national and provincial perspective. 

• At the level of systems support staff, in respect of the M&E Framework itself: 

With ongoing improvements in data tracking, analysis and communication assisting 
these role-players in their duties of supporting the City in ensuring appropriate, well- 
planned and soundly implemented initiatives - in line with GDS and IDP 
commitments. 

• From a governance perspective: With M&E information used to provide feedback 
to the various oversight structures within the City, to citizens and residents, and to 
communication platforms, for engagement with other stakeholders such as provincial 
and national departments - thereby enhancing accountability and transparency. 

It is also useful to reflect on the City’s current supporting structures, systems, processes, 
roles, responsibilities and reporting arrangements associated with performance management 
- for the sake of completeness. It is in this context that M&E occurs, with those responsible 
and accountable for delivery being key users of M&E information. The table below illustrates 
the various areas of responsibility and accountability of role-players within the City’s context, 
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with input adapted to address the unique specificities of the City. It builds on National 
Treasury’s MFMA Circular (2006, p.1) in respect of oversight (Circular No. 32), which notes 
that “While, in the first instance it is left to the mayor or the Executive Committee to resolve 
any performance failures, ultimately the Council is vested with the power and responsibility 
to oversee both the executive and administration. Oversight occurs at various levels in the 
municipality...’’. Understanding the nature of responsibility, accountability and oversight 
across the City aids in building a clearer sense of those role-players for whom M&E 
information holds critical value. 


Table 6.1: Governance framework applicable to local government (Adapted from 
National Treasury, 2006, p. 1) 


Role-player 

Responsible for: 

Oversight over: 

Accountable to: 

Speaker 

Ensuring smooth 
functioning of Councii 

EM 

The Community 

Council 

Approving strategy, poiicy 
and budget 

EM or Committee 

The Community 

Cluster 

Mayoral Sub- 
committee 
(i.t.o. each 
cluster) 

Reviewing and giving input 
to Ciuster deiivery and 
proposais 

Ciuster and reiated 
MMC 

Chairperson of Committees 

EM or 
Committee 

Strategy, poiicy, budgets, 
outcomes, management of/ 
oversight over municipai 
manager 

CM 

Councii 

MMCs 

Strategy, poiicy, budgets, 
outcomes, management of/ 
oversight over Portfoiio and 
reiated Ciuster 

Portfoiio and reiated 
Ciuster 

EM; The Ciuster Mayorai 
Sub-Committee 

CM 

CoJ Group outputs and 
impiementation 

The Administration - 
EDs, MDs and CEOs 

EM or Committee 

Boards of 
MOEs 

Approving strategy, poiicy 
and budget in respect of 

ME 

The Administration - 
EDs, MDs and CEOs 

Councii 

EDs, MDs and 
CEOs 

Directorate/ ME specific 
outputs and impiementation 

Operations in respect 
of Directorate or ME 

Portfoiio/ Ciuster MMC (i.t.o. 
oversight) and CM (day to 
day management) 


The figure below elaborates further on these arrangements, drilling down in terms of the 
various roles of the Executive - vis-a-vis oversight, decision-making and delivery - and 
thereby clarifying the nature of M&E information required. 
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Figure 6.1: Role-players in the Executive - Oversight, decision-making and delivery 


EXECUTIVE 


LEGISLATURE 

(Legislative, participatory) 



Mayoral Sub-Committees 

Coordination and integration of 
programmes 

Service delivery monitoring 
(programme based) 

Pre-screening of reports, and advice 
on reports requiring Mayoral 
Committee deliberation 


It is critical that frameworks such as the M&E Framework outlined here are aligned with the 
City’s institutional form. This is acknowledged as a requirement for ease of implementation. 
Further input on the practical ways in which alignment will take place between the steps 
included in the M&E Framework, and the various role-players across the City’s various 
governance and delivery structures, will be provided within the M&E Handbook that is to 
follow. 
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7. Next steps in implementing the M&E Framework 


Implementing an M&E Framework is a long-term exercise, with inputs required to ensure the 
development of an appropriate approach and supporting mechanisms. To drive successful 
implementation, a range of tools, systems and resources need to be put in place. Success is 
also heavily dependent on the type of culture the City is able to grow in relation to M&E - 
with a focus on continual learning necessary. This chapter addresses the next steps for full 
implementation of the M&E Framework - setting the foundation for the path ahead. 


7.1 . Requirements for implementation 

The table below presents an overview of the necessary conditions for the M&E Framework 
to be rolled out effectively across the City. Included is an overview of the elements needed, 
and the actions required to ensure their delivery. Ideas have been drawn from various 
literature sources (e.g. Kusek & Rist, 2004; AudAID, 2005; DPME, 2011; Lahey, 2009), the 
City’s own experiences and learnings reflected in Chapter 3. 


Table 7. 1: Requirements for implementation -and steps necessary for delivery 


Requirement 

Focus 

Steps required tor delivery 

Formalise the 
M&E system 

Ensure a common 
understanding of 
the M&E system 
and its 
requirements 

• Review and refine the M&E Framework, to ensure 
appiicabiiity to the City and its stakehoiders. 

• Deveiop a City-wide poiicy and procedure (e.g. an M&E 
Handbook) to ensure sound impiementation of the M&E 
Framework. 

Build and grow 
a demand for 

M&E results 

Ensure widespread 
and ongoing 
demand for data 
and M&E findings - 
across aii ieveis of 
the City (thereby 
ensuring quaiity 
coiiection by aii 
members of the 

City - regardiess of 
roie or ievei). 

• Roii-out a communication and education campaign 
amongst its empioyees, for aii to understand their roie in 
reiation to the City’s overarching vision and goais - e.g. 
transiating the City’s iong-term CDS into an image of 
what every City empioyee and every stakehoider within 
the city can do to support deiivery. 

• Train aii empioyees, for aii to understand their roie in 
data coiiection and appiication. 

• Drive a consistent ieadership message regarding the 
need for rigorous M&E practice across the CoJ. 

Encourage 
alignment with 
principles of 
accountability 
and 

transparency - 

ensuring 

credibility 

Ensure aii within 
the City are heid to 
account for deiivery 

• Commit to and undertake ongoing engagement and 
communication with stakehoiders, in respect of pians, 
deiivery, chaiienges, M&E findings, and potentiai 
soiutions. 

• Pubiish evaiuation findings on the organisation’s 
website; and inciuding comment and input on M&E 
findings within annuai reports and other organisationai 
documents - regardiess of the favourabiiity of findings. 

Build an M&E 

focused 

culture 

Estabiish an 

outcomes-oriented 

cuiture within the 

• Ensure communication at reguiar stages throughout the 
M&E cycie, to buiid an understanding of activities 
underway, access to findings, stakehoider engagement 
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Requirement 

Focus 

Steps required for delivery 


City - encouraging 
a focus on accurate 
data, anaiysis, 
iearning, 
appiication of 
insights and 
ongoing 
stakehoider 
engagement 
regarding deiivery 

and participation, commitment to the system, and 
appiication of iearnings. 

• Aiign rewards with an outcomes approach - focusing 
beyond activities, where appiicabie, to other ieveis (e.g. 
aiignment with City-wide outcomes / impacts). 

• Impiement change management activities that 
communicate that aii are responsibie - in iine with the 
perspective refiected in the Joburg 2040 CDS: “We are 
aii piayers”. 

• Consider other mechanisms through which to drive in 
an outcomes-based cuiture - e.g. a Code of Conduct or 
Behaviours Charter, with aiigned incentives. 

Establish clear 

roles and 
responsibilities 

Deveiop ciarity on 
roies and 
responsibiiities 
reiating to the M&E 
Framework 

• Confirm roies and responsibiiities with regards both 
monitoring and evaiuation-reiated activities (e.g. in 
terms of: process estabiishment; roii-out; quaiity controi; 
training; data coiiection, anaiysis for monitoring versus 
evaiuation, reporting, storage, management and 
oversight) - and communicate through a ciear matrix. 

• Deveiop and communicate a formai poiicy document, 
documenting roies and responsibiiities - to ensure a 
shared understanding. 

Build adequate 
M&E capacity 

Estabiish adequate 
capacity to design, 
impiement, support 
and institutionaiise 
the M&E 

Framework 

• Create suitabie partnerships with externai stakehoiders, 
technicai speciaiists and M&E partners. 

• Identify capacity gaps (across manageriai, technicai, 
administrative, counseiior, communication and other 
roies) - and address through capacity buiiding. 

• House the technicai responsibiiity for oversight of 
impiementation of the M&E Framework with a defined 
roie-piayer - with this most probabiy being the GSPR. 

Establish 

supporting 

M&E systems 

Deveiop, maintain 
and improve an 

M&E system 
(addressing 
indicator, data and 
anaiysis 
information) 

• Design and estabiish an M&E system, to aiiow for easy 
data coiiection, storage and retrievai - and ongoing 
knowiedge management. 

• Ensure M&E system can iink with other internai systems 
and knowiedge management frameworks. 

• Support the M&E system with an up to date indicator 
and data-base framework. 

Ensure the 
production of 
credible and 
trustworthy 
information 

Ensure data is 
vaiid, reiiabie, 
credibie, 
transparent, 
understandabie and 
verifiabie 

• Review and assess current data coiiected - and identify 
mechanisms to improve quaiity and quantity, where 
necessary. 

• Estabiish partnerships for M&E data coiiection, anaiysis 
and reporting, and access to existing sources. 

• Estabiish an M&E Centre for Exceiience - responsibie 
for the data management and maintenance system. 

• Ensure evaiuator independence - e.g. through stressing 
neutraiity of evaiuators and M&E function; ensuring 
evaiuators have ciear access to management and 
ieadership; estabiishing an Evaiuation Committee 
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Requirement 

Focus 

Steps required for delivery 



comprised of senior role-players. 

• Identify if additional systems or tools are required for 
sound data collection and data credibility - and if so, 
develop/ establish/ procure. 

Align with 
other reporting 
and delivery 
mechanisms 

Build on wider 

value of M&E 

Framework - 
ensuring alignment 
with internal/ 

external needs 

• Ensure M&E plans align with the long-term GDS - with 
operational and M&E planning cascaded from the GDS. 

• Align M&E content with internal management 
requirements - and the City’s obligations in respect of 
reporting to National Government. 

Ensure 
application of 
M&E findings 

Maintain 

awareness that 

data is not an end 
in itself - with value 
lying in application 
and learning 

• Establish mechanisms for linking information back to 
decision-making and managerial roles and role-players. 

• Focus training and management on building capacity 
and interest in gathering and analysing information - 
and applying insights. 

Focus on 

continuous 

improvement 

Ensure the M&E 

Framework and 
supporting systems 
evolve - improving 
with use 

• Establish quality control mechanisms and activities - 
e.g. periodic involvement of the AG in evaluations of the 
M&E system’s efficacy; evaluation of how effectively 

M&E information has been applied; analysis of areas for 
improvement. 


While the above is indicative of the many steps necessary before the City has an M&E 
system in place that is fully aligned to the outcomes approach, there is much work that has 
already been done. In its effort to constantly improve internal operations and short, medium 
and long-term delivery to citizens and residents, the City acknowledged the steps necessary, 
and has committed itself to the challenge. 


7.2. Concluding comments 

This M&E Framework has focused on establishing a shared understanding of the domain 
within which M&E occurs, clarifying concepts and the mechanisms in use within the other 
spheres of government, and by those external to government - both within and beyond 
South Africa’s borders. The City’s planning, delivery, M&E and reporting processes, systems 
and tools have been explored, with shifts outlined, to enable alignment with the outcomes 
approach to M&E. Good practice principles in relation to M&E systems have been explored, 
along with accepted practices in terms of M&E implementation. 

The above has been used as a basis for articulating an M&E Framework for the City - 
founded on the insights established. This represents the first stage of our journey in 
designing and implementing a more robust approach to M&E. Ultimately, this framework is 
primarily concerned with ensuring a real focus on the delivery of the City’s long-term 
outcomes - through short and medium-term planning, delivery, monitoring and evaluation 
mechanisms, and the associated reporting. 

The City has identified a number of benefits it aims to see emerging through implementation 
of the M&E Framework. Based on research and experiences in other areas, it is anticipated 
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that, if implemented and used appropriately, the envisaged M&E Framework will provide the 
City with: 


• A sound foundation for planning - and a mechanism to aid decision-making relating 
to resource usage, activities, projects and programmes; 

• A framework through which to gather meaningful information that supports reflection, 
learning, adaptation and/or improvements; 

• An approach through which to identify issues or challenges timeously, where these 
may otherwise hamper delivery - and to adjust plans accordingly; 

• A tool through which to assess delivery against plans, and to build greater 
understanding of causality and assumptions in project delivery, to support future 
replications of effective service provision; 

• The means to support informed communication and engagement with stakeholders; 

• A mechanism to enhance accountability in terms of the manner in which resources 
and public funds are put to use; 

• A system through which to promote the principle of public participation - with input 
from members of the public aiding identification of challenges, new needs and/ or 
solutions, thereby supporting improved delivery; and 

• Ultimately, a mechanism to improve the achievement of outcomes defined within the 
City’s strategic plans - including those defined within the Joburg 2040 GDS - 
increasing the achievement of developmental objectives. 
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AG 

AusAID 
CEO 
The City 
The city 

CM 

CoJ 

CoJ Group 

CoGTA 

GSPR 

DG 

DPME 

DPSA 

ED 

EM 

EMT 

FMPPI 

GDS 

GPG 

GWMES 

IDP 

Joburg 

JPAC 

JR AS 

M&E 

MD 

MDG 

ME 

MFMA 


Annexure 1 : Acronyms and abbreviations 

Auditor General 

Australian Agency for International Development 

Chief Executive Officer 

The Metropolitan City of Johannesburg 

The geographic area that falls within the demarcated boundaries of 
Johannesburg - and the area for which the City of Johannesburg is 
responsible in its role as local government. 

City Manager 

The City of Johannesburg 

The City of Johannesburg Group (including all core internal 
departments and all Municipal Entities) 

Department of Cooperative Governance and Traditional Affairs 

Group Strategy Policy and Relations 

Director General 

Department of Performance Monitoring and Evaluation 

Department of Public Service and Administration 

Executive Director 

Executive Mayor 

Executive Management Team 

Framework for Managing Programme Performance Information 

Growth and Development Strategy 

Gauteng Provincial Government 

Government-wide Monitoring and Evaluation System 

Integrated Development Plan 

The City of Johannesburg 

Johannesburg Performance Audit Committee 

Johannesburg Risk Assurance Services 

Monitoring and Evaluation 

Managing Director 

Millennium Development Goal 

Municipal Entity 

Municipal Finance Management Act 
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MMC 

Member of the Mayoral Committee 

MTSF 

Medium-Term Strategic Framework 

NDP 

National Development Plan 

NEPF 

National Evaluation Policy Framework 

NGO 

Non-governmental Organisation 

NPC 

National Planning Commission 

SDBIP 

Service Delivery and Budget Implementation Plan 

SHU 

Shareholders Unit 

Stats SA 

Statistics South Africa 

ToR 

Terms of Reference 

UN 

United Nations 

UNDP 

United Nations Development Programme 

WFP 

World Food Programme 
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Annexure 2: Proposed M&E roles and responsibilities matrix 


Role-player 

Role 

Legislative arm of 
Council 

• Signs-off on M&E Policy and Framework 

• Approves City strategies, plans and budgets in respect of M&E 

• Carries out oversight duties in terms of delivery across the CoJ Group, 
supported by M&E data and Framework 

• Holds the City’s Executive and leadership accountable for delivery, with the 
Council serving as ultimate oversight of the CoJ Group 

• Drives the M&E system - demonstrating leadership in being accountable and 
transparent 

• Oversees implementation of the principle that M&E findings are addressed 
through prompt action / changes at the appropriate level 

• Engages with the community and electorate on M&E findings - in support of 
community oversight 

Mayoral Sub- 
committee - 
Cluster specific 

• Reviews performance and delivery in respect of cluster as per oversight role, 
and carries out fiduciary duties and duties of care in this regard - making use 
of M&E system and analysis 

• Uses monitoring activities to support delivery, in areas of responsibility 

• Ensures appropriate independent evaluations are performed, as needed, in 
support of performance improvement and oversight 

• Oversees implementation of the principle that M&E findings are addressed 
through prompt action / changes at the appropriate level - in respect of cluster 

Executive arm of 
Council 

• Reviews, recommends improvements to, and confirms M&E Policy and 
Framework for submission to Council 

• Reviews, recommends improvements to and confirms City strategies, plans 
and budgets in respect of M&E - for submission to Council 

• Drives the M&E system - demonstrating leadership in being accountable and 
transparent (with M&E used for all stages of the planning, delivery, and 
performance improvement cycle) 

• Reviews CoJ Group performance and delivery as per oversight role, , 
alongside the Performance Evaluation Panel, and carries out fiduciary duties 
and duties of care in this regard, supported by M&E data and Framework 

• Uses monitoring activities to support delivery, in areas of in areas of 
responsibility 

• Ensures appropriate independent evaluations are performed, as needed, in 
support of performance improvement and oversight 

• Oversees implementation of the principle that M&E findings are addressed 
through prompt action / changes at the appropriate level - while taking action 
in those areas of responsibility 

• Engages with the community and electorate on M&E findings - in support of 
the concept of community oversight 


The City of Johannesburg’s Monitoring and Evaluation Framework 1 4 May 201 2 

80 I Page 


Role-player 

Role 

CM - as 

Accounting 

Officer 

• Reviews, recommends improvements to, and confirms M&E Poiicy and 
Framework for submission to Mayorai Committee (as head of EMT) 

• Reviews, recommends improvements to and confirms City strategies, pians 
and budgets in respect of M&E - for submission to Mayorai Committee 
(foiiowing review via EMT) 

• Drives the M&E system - demonstrating ieadership in being accountabie and 
transparent (with M&E used for aii stages of the pianning, deiivery, and 
performance improvement cycie) 

• As Accounting Officer, ensures rigour of the M&E Framework - in terms of 
poiicy, procedure and supporting requirements, and the quaiity and integrity of 
M&E information and supporting systems 

• Reviews CoJ Group performance and deiivery as Accounting Officer, 
aiongside the Performance Evaiuation Panei - with support of M&E data and 
Framework 

• Uses monitoring activities to support deiivery, in areas of in areas of 
responsibiiity 

• Ensures appropriate independent evaiuations are performed, as needed, in 
support of performance improvement and oversight 

• Oversees impiementation of the principie that M&E findings are addressed 
through prompt action / changes at the appropriate ievei - whiie taking action 
in those areas of responsibiiity 

Head of internai 
department/ ME 

• As part of EMT, reviews and recommends improvements to M&E Poiicy and 
Framework for submission to Mayorai Committee 

• Ensures impiementation of M&E Framework within area of accountabiiity - 
demonstrating ieadership in driving impiementation, and in being accountabie 
and transparent in respect of M&E system set-up and findings 

• Reviews performance and deiivery in area of accountabiiity - with support of 
M&E data and Framework 

• Uses monitoring activities to support deiivery, in areas of in areas of 
responsibiiity 

• Ensures appropriate independent evaiuations are performed, as needed, in 
support of performance improvement and oversight 

• Ensures M&E findings are addressed through prompt action / changes at the 
appropriate ievei - whiie taking action in those areas of responsibiiity 

• Oversees impiementation of the principie that M&E findings are addressed 
through prompt action / changes at the appropriate ievei - whiie taking action 
in those areas of responsibiiity 

Director/ Deputy 
Director/ iine 
manager/ 
supervisor 

• Ensures impiementation of M&E Framework within area of accountabiiity - 
demonstrating ieadership in driving impiementation, and in being accountabie 
and transparent in respect of M&E system set-up and findings 

• Sets performance indicators and targets as part of annuai performance 
pianning 

• Deveiops annuai M&E Pian for each project or sub-programme 
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Role-player 

Role 

• Reviews performance and delivery in area of accountability - with support of 
M&E data and Framework 

• Uses monitoring activities to support delivery, in areas of responsibility 

• Ensures appropriate independent evaluations are performed, as needed, in 
support of performance improvement 

• Ensures M&E findings are addressed through prompt action / changes at the 
appropriate level - while taking action in those areas of responsibility 

• Reports M&E findings in quarterly performance reports and annual reports 

• Shares experiences and knowledge, in support on ongoing learning from M&E 

Employees within 
the City 

• Actively provide feedback into the M&E system, to ensure up to date and 
relevant data 

• Set performance indicators and targets as part of annual performance 
planning - for areas of responsibility 

• Participate in reviews of performance and delivery in area of responsibility - 
with support of M&E data and Framework 

• Use monitoring activities to support delivery, in areas of responsibility 

• Ensure M&E findings are addressed through prompt action / changes within 
area of responsibility 

• Share experiences and knowledge with colleagues, in support on ongoing 
learning from M&E 

M&E Centre of 
Excellence/ 
“Strategy 
Implementation 

Unit” (may be 
within the 
Performance 
Management, 
Monitoring and 
Reporting Unit) 

• Establishes M&E policy and guidelines 

• Ensures establishment of a suitable M&E system (including IT capabilities) 

• Provides M&E capacity building support/ training -and builds M&E capacity 
within the CoJ Group 

• Provides technical assistance 

• Ensures implementation of the framework, and supports with implementation 
assistance 

• Provides support for structured learning opportunities and moments of 
reflection 

• Establishes mechanisms through which to test how much recommendations 
and data from M&E are implemented 

• Reports on M&E Framework implementation and recommends updates as 
needed 


The City of Johannesburg’s Monitoring and Evaluation Framework 1 4 May 201 2 

82 I Page 


Annexure 3: Organisational, cluster, departmental or project 
planning, in the context of the outcomes approach 


The success of M&E within the City hinges on the establishment of a shared sense of 
purpose amongst all players within the City, and a commonly held view of the applicable 
‘theory of change’ - that is, “the causal mechanisms between the activities, outputs, 
outcomes and impacts” (DPME, 2011, p.4). Establishing a shared theory of change requires 
a clear and consistent view of the following - phrased as steps within the operational 
planning cycle - in contrast with the M&E cycle addressed in Chapter 5 above: 

• The desired long-term impacts the City hopes to achieve (Step 1) 

• Appropriate outcomes aligned to the identified impacts - i.e. those outcomes that 
will support achievement of the impacts (Step 2) 

• Outputs identified as necessary to deliver on the desired outcomes (Step 3) 

• Those activities that will lead to the defined outputs (Step 4) 

• The various inputs required, for delivery on the defined activities (Step 5) 

Steps 1 and 2 generally relate to strategic planning (as reflected within the GDS, and the 
various IDPs through which this is operationalized), while steps 3, 4 and 5 tend to align more 
with the City’s ‘business planning’ and annual planning processes. 

Box 5.3: Establishing the ‘theory of change’ - the foundation for Steps 1 to 5 


Identification of an appropriate and sound theory of change depends on the extent to which 
a clear understanding exists of the status quo, and the ‘problem’ that interventions must 
address. To ensure that ideas relating to the theory of change are made explicit - and are 
therefore opened up for debate and confirmation - the City proposes that use is made of a 
tool such as the ‘problem tree’ approach, as outlined in Chapter 2. By portraying the 
understanding held of underlying constraints, causes and effects within each area of 
analysis, it will be possible to identify suitable solutions more readily. A set of guidelines 
further elaborating on development of ‘problem trees’ will be included in the M&E Handbook. 

The emerging problem tree analysis serves as a base for the identification of solutions, 
during which focus should be placed first on the end goal (i.e. the impact envisaged) - and 
then an articulation of the outcomes, outputs, activities and inputs needed to achieve this. 
Each of these elements is covered in more detail below, as part of the step-by-step process 
of M&E planning. In effect, steps 1 to 5 will result in the formulation of a theory of change. 
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step 1 : Identify the envisaged long-term impact [‘what we aim to change’] 


Step 1 - identification of the envisaged 
long-term impact - falls within the domain 
of organisational delivery. This is the 
starting point of all organisational 
planning, including - ultimately - the 
planning for M&E activities. As illustrated 
alongside, ‘impacts’ link strongly with 
planning at the level of the Joburg 2040 
GDS - with achievement only likely in the 
long term. 

Many factors influenced the goals or impacts included in the City’s long-term GDS. Amongst 
these are national, provincial and local government priorities, events beyond the national 
border, legislation, prior City strategies, research, and stakeholder input and insights gained 
through City’s GDS Outreach process. While the impacts identified in the GDS serve as the 
critical starting point, M&E requires an up to date reflection on where the City aims to be - 
taking into account its current circumstances, and the reality of those stakeholders who live 
or work within its boundaries. Identifying envisaged impacts therefore involves: 

• A status quo analysis, and research into the future vision for the City - with due 
consideration of the following areas, amongst others: the GDS and other aligned 
long-term plans, challenges and opportunity faced, shaping forces, and dynamics 
within other spheres of government; 

• A problem analysis - to identify the gaps between the desired future, the current 
state and the causal steps to support a movement to this future (as noted above); 

• Testing the future vision and the desired impacts through participatory processes 
such as the GDS Outreach; 

• Refining the defined impacts, on the basis of thorough analysis of information 
emerging from the research and stakeholder engagement process; and 

• Recognising the need to review ‘impacts’ on a reguiar basis, ensuring continued 
applicability in the context of a rapidly changing environment. 

Are impacts set in stone, once finalised? No strategy is static. The benefit of an M&E system 
is that, through the process of gathering, analysing and considering the application of real- 
time information, red flags may emerge in respect of any of the levels of the framework - i.e. 
in terms of impacts, outcomes, outputs, activities, or inputs. Leaders and managers are 
accountable for listening to, considering and where appropriate, applying new insights to 
build a revised theory of change. 
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step 2: Identify outcomes for desired impacts [‘what we wish to achieve’] 


Step 2 of operational planning process 
relates to the identification of those 
outcomes that will contribute to the 
achievement of the goals or desired 
impacts identified in Step 1 . While goals or 
‘impacts’ tend to be long-term in nature, 

‘outcomes’ are often focused on the long to 
medium-term period, aligning with the 
planning period associated with the City’s 
GDS, five-year IDP or Cluster plans. 

In identifying outcomes: 

• The theory of change should be 
appropriateness of outcomes tested, in the context of the envisaged impact; 

• Outcomes should be identified through an inclusive, participatory process - with 
the underlying logic and assumptions reviewed, debated and through this process, 
jointly-owned by all stakeholders; 

• Potential outcomes emerging through the above shouid be prioritised, with due 
consideration of stakeholder concerns and the realities of limited resources. Focus 
should also be placed on identifying the best possible approach through which to 
achieve outcomes - with it noted that “One budgets to outputs and manages to 
outcomes” (Kusek & Rist, 2004, p.57); 

• Outcomes are positive present-tense statements of the changed state, following 
its achievement. They should include one area of improvement only - e.g. “All 
people in South Africa feel and are safe” (The Presidency, 2010, p.13). 

Impacts and outcomes are largely applicable to the City as an organisation, or to clusters, 
departments or entities. The elements that follow (outputs, activities and inputs) are more 
applicable to performance driven within a shorter timeframe (e.g. a year), by clusters, 
departments and individuals. 


Levels of Performance Indicators Planning Levels 



made explicit, and interrogated - with the 


Step 3: Identify outputs linked to outcomes [‘what we produce or deliver’] 


This step involves the identification of 
those outputs that will contribute to the 
achievement of the defined outcomes. As 
reflected alongside, outputs tend to be 
framed within the context of short to 
medium-term delivery - reflected in 
planning mechanisms such as the SDBIP, 
business plans, one-year cluster plans, or 
within individual scorecards. 


Levels of Performance Indicators 


Planning Levels 



The City of Johannesburg’s Monitoring and Evaluation Framework 

85 I Page 


14 May 2012 




In developing outputs: 

• Prioritisation is again important, in the context of resource constraints; 

• It may be necessary to identify a number of outputs for the achievement of each 
outcome, with these outputs potentially staggered across a range of time-periods; 

• Focus should be placed on what is produced or delivered, with each output stated in 
the past tense, reflecting what has been achieved, provided or produced (e.g. 
“Children immunized”); 

• Different types of outputs should be identified - with a balanced focus on outputs 
that fall within the three categories identified within Chapter 2 - i.e.: ‘continuous 
improvements, ‘impactful improvements’, and ‘broad initiatives’. Some of these 
may be achievable in the short-term, while others will require a lengthier period of 
delivery. 

Step 4: Identify activities linked to outputs [‘what we do’] 

Activities are “a collection of functions 
(actions, jobs, tasks) that consume inputs 
and deliver benefits and impacts” (The 
Presidency, 2010, p.11). While some 
individual scorecards and annual business 
plans may focus on outputs (and potentially 
outcomes), many will only cater for activities 
in the context of a year-long planning 
period. In these cases, outputs may require 
more than a year of action, before they can 
be achieved. 

When identifying activities, it is noted that: 

• These should be stated in the present tense, and should contain a verb - e.g. 
“Provide advice. . .” or “Conduct training. . .”; 

• Focus should be placed on reviewing activities in the context of associated inputs, 
and those outputs and outcomes they are meant to contribute towards - ensuring 
alignment. 

Step 5: Identify inputs required [‘what we 

Step 5 involves identification of those 
resources required to carry out a particular 
activity to the defined level. Inputs are 
generally considered in the context of a 
short to medium-term planning period - e.g. 
within the annual planning cycle. It should 
however be noted that they may also need 
to be planned in the context of multi-year 
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Levels of Performance Indicators 


Planning Levels 


Impacts (Goals) 

!=> 

GDS 

Outcomes 

>=> 

GDS/ (DP/ Cluster plans 

Outputs 

! 

1 


Activities 


SDBIP/Cluster'piBrTsK 
Business Plans/ 
Scorecards 

Inputs 

1 

1 

1 
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projects, with learnings in one cycle in respect of outcomes, outputs, activities or inputs 
applied back to forecasts relating to inputs. Inputs may vary significantly from one activity to 
the next - and may include a wide variety of elements such as funding, people, information 
technology, materials, infrastructure or tools. 


Bringing it together: Steps 1 to 5 represented through an outcomes ‘chain’ 

The table below provides a view of how Steps 1 to 5 may be integrated into a table of 
planned delivery - clearly outlining the applied theory of change (i.e. demonstrating the 
causal links envisaged between inputs, activities, outputs, outcomes and impacts, across a 
planning timeframe). This represents a ‘results’ or outcomes chain. Also depicted is the 
concept of ‘working backwards’ - from the impact envisaged, through to each of the 
elements needed to realise this, with some outcomes and outputs staggered across the 
timeframe. A template for the development of an outcomes chain is included in Annexure 4. 
These issues will be addressed more fully within a revision of the City’s Performance 
Management Framework. 


Table 1: An outcomes chain, integrating steps 1 to 5 (Template in Annexure 4) 


Impact 
(What we 
aim to 
change) 

Outcomes 
(What we wish to 
achieve) 

Outputs 

(What we produce or 
deiiver) 

Activities 
(What we do) 

Inputs 

(What we use to do the 
work) 

Joburg is 
food secure 
by 2040 

By 2030, 
nutritionai intake 
of most vuinerabie 
groups with no 
support other than 
government is 
maintained at 
optimai ieveis 

2030: Net food gap of 
vuinerabie peopie met 
through state suppiy 
(20%) and househoid 
food project initiatives 
(80%) - enabiing seif- 
dependency 

• Identify househoids in need 

• Train househoids to grow 
vegetabie gardens 

• Provide periodic mentoring 
and suppiy of materiais 

• Prepare food packs 

• Distribute food to community 
centres 

• Training materiais 

• Seed and equipment 

• Funding 

• Food 

• Distribution support 

• Staff 



2020: Net food gap of 

15 000 vuinerabie peopie 
(women, chiidren, peopie 
with disabiiities, 
househoids beiow the 
poverty iine) met through 
sustainabie state suppiy 

• Identify vuinerabie persons 

• Prepare food packs 

• Distribute food to community 
centres 

• Food 

• Funding 

• Distribution support 

• Staff 


By 2040, urban 
agricuiture 
provides 35 % of 
iocai food needs 

2030: 100 smaii scaie 
community-initiated and 
driven projects 
operationai for at ieast 5 
years 

• Provide ad hoc mentoring 
support 

• Estabiish support network 

• Deveiop iearning piatform 

• Trained mentors 

• Meeting venue 

• Communication materiai 

• Database of projects 



2018: Start-up materiais 
and mentoring provided to 
aii trainees for first year of 
operation 

• Identify materiais required 

• Source materiais 

• Distribute materiais 

• Conduct periodic mentoring 

• Seed 

• Compost 

• Insecticides 

• Impiements 

• Skiiied mentors 



2015: 200 peopie trained 
in urban agricuiture 
techniques 

• Prepare for training 

• Conduct training 

• Carry out evaiuation 

• Technicai speciaiists 

• Training materiais 

• Training venue 

• Evaiuation panei 
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Annexure 4: Templates - An outcomes chain 


Impact 

(What we aim 
to change) 

Outcomes 

(What we wish to 
achieve) 

Outputs 

(What we produce or 
deliver) 

Activities 

(What we do) 

Inputs 

(What we use to 
do the work) 
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Annexure 5: Templates - Indicator planning framework 


Indicator 

Data 

source 

Data 

collection 

method 

Who will 
collect the 
data? 

Frequency 

of 

collection 

Cost and 
difficulty to 
collect 

Who will 
analyse the 
data? 

Who will report 
data? How often? 

Report / 
feedback 
format? 

Who will use the 
data? (Audience and 
interests) 

Description of 
Indicator 1 










Description of 
Indicator 2 










Description of 
Indicator 3 










Description of 
Indicator 4 










Description of 
Indicator 5 
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Annexure 6: Templates - A City-wide M&E Plan (timeframe specific) 


Impact: [Description of impact/ goai 
at a City] 

Institutional metric 

Institutional annual targets 

Implementing 

Cluster 

Institutional 

Budget 

Pianning cycie: E.g. 2020 
-2025 

Key 

outputs 

lndicator/s° 

Means of 
Verification 

5-year 

IDP 

Target 

Baseline 

Key outcomes 

Yr 

-3 

Yr 

-2 

Yr 

-1 

Current year target 

Yr 

1 

Yr 

2 

Yr 

3 

Yr 

4 

Yr 

5 

Yr 

1 

Yr 

2 

Yr 

3 






















































































































































































^ This could also be structured to include target information - i.e. indicator and exact target. 
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Annexure 7: An example of the M&E Plan in action 


The figure below provides a view of the GDS’ Outcome #2: “A resilient, liveable, sustainable 
urban environment - underpinned by infrastructure supportive of a low carbon economy”. 
Four outputs are detailed in the GDS, including: 

• Sustainable and integrated delivery of water, sanitation, energy and waste 

• Eco-mobility 

• Sustainable human settlements 

• Climate change resilience and environmental protection 

This provides a valuable example of M&E in practice. The benefit of a sound M&E system is 
that it allows for reflection, and revision of defined goals, objectives, outputs and activities 
where required, on the basis of learning and application. Through making the theory of 
change and underlying assumptions explicit, refinements can be made. 

In this case, where reflecting on the outputs described, it is clear that they are more closely 
aligned with the concept of an outcome or sub-outcome. The statement for Outcome 1 may 
also be more closely reflective of the long-term impact - and may be better structured in this 
way (with appropriate focus on ensuring one idea is included therein, as per good practice). 
A further level of detail is required in terms of the outcomes, to ascertain outputs - and the 
associated activities and inputs. With the focus on stating these outcomes in more detail, to 
ensure clarity, they may be constructed as follows: 

• Become more energy efficient and diversify the city’s energy mix 

• Re-cycle and re-use the city’s scarce water resources 

• Reduce, re-cycle and re-use the city’s waste 

• Create a more compact city form with good quality housing and shelter 

• Shift mobility to public transport and environmentally friendly fuel sources 

Viewing these outcomes from an outcomes-based approach, in the context of a future 2040, 
the following image (presented as a conceptual representation) of associated outputs, sub- 
outputs, activities and inputs may emerge - in relation to the outcome of: “Re-cycle and re- 
use the city’s scarce water resources”. 
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The table below provides an example of some of the elements applicable within the M&E Plan - in alignment with the above depiction - with 
reframed impacts, outcomes and outputs. This lays the foundation for identification of MoV, and the applicable baselines, targets, assumptions 
and reporting arrangements. 


Impact: Joburg is water secure 

City-wide metric 

City-wide annual targets 

Implementing Cluster 

City-wide Budget 

Planning cycle: 
Eg. 2020- 
2025 

Key outputs 

Indicator/s^ 

Means of 
Verificatio 

n 

5-year 

IDP 

Target 

Baseline 

Key outcomes 

Yr 

-3 

Yr 

-2 

Yr 

-1 

Current year target 

Yr 

1 

Yr 

2 

Yr 

3 

Yr 

4 

Yr 

5 

Yr 

1 

Yr 

2 

Yr 

3 

Re-cycle and 
re-use the city’s 
scarce water 

resources 

Service 
provider/ 
specialist in 
acid mine 
drainage 
appointed 

• Percent of 
water 
reclaimed 

• Percent of 
community 
engaged in 
water 
recycling 

• Signed 
Service 

Level 

Agreement 















































































































































^ This could also be structured to include target information - i.e. indicator and exact target. 
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Annexure 8: Templates - M&E Plan at the cluster level 


Impact: [Description of impact/ goai 
at a Cityievei] 

City-wide metric 

Cluster-specific annual targets 

Assumptions, 
risks and 
mitigation 

Implementing 
department/ 
entity/ section 

Implementing 
department/ entity/ 
section budget 

Pianning cycie: E.g. 
2020-2025 

Key 

outputs 

Indicator/s 

10 

Means of 
Verification 

5-year 

IDP 

Target 

Baseline 

Key outcomes 

Yr 

-3 

Yr 

-2 

Yr 

-1 

Current 
year target 

Yr 

1 

Yr 

2 

Yr 

3 

Yr 

4 

Yr 

5 

Yr 

1 

Yr 

2 

Yr 

3 
































































































































































































This could also be structured to Include target information - i.e. Indicator and exact target. 


The City of Johannesburg’s Monitoring and Evaluation Framework 


94 I Page 


14 May 2012 


Annexure 9: Template - Annual M&E Plan at the inter-cluster level 


Annual plan for: [Year] 

Cluster-specific metric 

Cluster-specific annual 
milestones 

Implementing 
department/ entity/ 
section 

Budget 

Key outcomes 

Key outputs 

Activities 

Indicator/s^' 

Means of 
Verification 

Baseline 

Annual 

Target 

Yr1 

Q1 

Q2 

Q3 

Q4 

Estimate 

Variance 
































































































































This could also be structured to Include target information - i.e. Indicator and exact target. 
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Annexure 10: Template - M&E Plan at the inter-cluster level 


Impact: [Description of impact/ goai] 

Planning cycle: E.g. 2020 - 2025 


Cluster 1 

Cluster 2 

Cluster 3 

Cluster 4 

Cluster 1 


Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 2’s 
areas of focus 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 3’s 
areas of focus 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 4’s 
areas of focus 

Cluster 2 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 1’s 
areas of focus 


Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 3’s 
areas of focus 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 4’s 
areas of focus 

Cluster 3 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster Ts 
areas of focus 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 2’s 
areas of focus 


Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 4’s 
areas of focus 

Cluster 4 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster Ts 
areas of focus 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 2’s 
areas of focus 

Inciude the contribution to be 
made - at the ievei of output or 
activity, in reiation to Ciuster 3’s 
areas of focus 
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Annexure 11 : Template - City-wide reporting template 


Impact: [Description of impact/ goai at a City 
ievei] 

City-wide metric 

Progress Report 

Budget Utilisation 

Comments 

Pianning cycie: E.g. 2020 - 2025 

Key 

outputs 

Indicator 

/S12 

MoV 

5-year 

IDP 

Target 

Baseline 

Annual 

target 

Quarterly 

targets 

Quarteriy 
Target 
Achieved 
(Yes/ No) 

if ‘Yes’, 
provide 
evidence 
if ‘No’, state 
intended 
mitigation 

Estimate 

Actual 

Variance 

Key outcomes 








Q1 







Q2 







Q3 







Q4 














Q1 







Q2 







Q3 







Q4 














Q1 







Q2 







Q3 







Q4 














Q1 







Q2 







Q3 







Q4 














Q1 







Q2 







Q3 







Q4 








This could also be structured to Include target information - i.e. Indicator and exact target. 
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Annexure 12: Template - Cluster-specific reporting template 


Impact: [Description of impact/ goai at a City ievei] 

City-wide metric 

Progress Report 

Budget Utilisation 

Comments 

Pianning cycie: E.g. 
2020-2025 

Key 

outputs 

Implementing 
department/ 
entity/ section 

Indicator 

/S'3 

MoV 

5-year 

IDP 

Target 

Baseline 

Annual 

target 

Quarterly 

targets 

Quarteriy 
Target 
Achieved 
(Yes/ No) 

if ‘Yes’, 
provide 
evidence 
if ‘No’, state 
intended 
mitigation 

Estimate 

Actual 

Variance 

Key outcomes 









Q1 







Q2 







Q3 







Q4 















Q1 







Q2 







Q3 







Q4 















Q1 







Q2 







Q3 







Q4 















Q1 







Q2 







Q3 







Q4 








This could also be structured to include target information - i.e. indicator and exact target. 
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